Denodo Virtual DataPort 4.6 Administration Guidehelp.denodo.com/platform-docs/4.6/DenodoVirtual... · Virtual DataPort 4.6 Administrator Guide. 7.6 EXPORT AS MICROSOFT WEB PART .....138

DENODO VIRTUAL DATAPORT 4.6 ADMINISTRATION GUIDE

Update Aug 16th, 2011

NOTE This document is confidential and is the property of Denodo Technologies (hereinafter Denodo). No part of the document may be copied, photographed, transmitted electronically, stored in a document management system or reproduced by any other means without prior written permission from Denodo.

Copyright © 2011 This document may not be reproduced in total or in part without written permission from Denodo Technologies.

Virtual DataPort 4.6 Administrator Guide

INDEX

PREFACE ............................................................................................................................................................................ I SCOPE .......................................................................................................................................................................... I WHO SHOULD USE THIS DOCUMENT ................................................................................................................... I SUMMARY OF CONTENTS....................................................................................................................................... I

1 INTRODUCTION...................................................................................................................................... 1

2 GENERAL ARCHITECTURE.................................................................................................................... 2 2.1 PHYSICAL LAYER................................................................................................................................... 3 2.2 LOGICAL LAYER...................................................................................................................................... 4

2.2.1 Data Module: Cache ................................................................................................................................. 5 2.3 USER LAYER............................................................................................................................................ 6

3 INSTALLATION AND INITIAL CONFIGURATION .............................................................................. 7

4 EXECUTION ............................................................................................................................................. 8 4.1 EXECUTING THE VIRTUAL DATAPORT GRAPHIC ADMINISTRATION TOOL .............................. 8 4.2 DOCUMENTATION AND ONLINE HELP ........................................................................................... 10 4.3 MAIN AREAS OF THE ADMINISTRATION TOOL ............................................................................ 10 4.4 TOOL PREFERENCES ........................................................................................................................... 11

4.4.1 Locale ...................................................................................................................................................... 11 4.4.2 Connection Parameters........................................................................................................................... 11

4.5 INVALIDATE CACHE ............................................................................................................................ 12 4.6 SEARCH ................................................................................................................................................. 12

4.6.1 Quick Search ........................................................................................................................................... 12 4.6.2 Catalog Search........................................................................................................................................ 12

4.7 QUERY MONITOR................................................................................................................................. 13

5 VISUAL CREATION OF VIEWS ........................................................................................................... 14 5.1 ACCESS TO THE GRAPHIC ADMINISTRATION TOOL.................................................................... 14 5.2 CREATING AND MANAGING PROJECTS ........................................................................................ 15

5.2.1 Moving Elements to Other Projects ........................................................................................................ 15 5.3 IMPORTING DATA SOURCES AND CREATING BASE VIEWS...................................................... 16

5.3.1 Importing JDBC-Type Sources ................................................................................................................ 16 5.3.2 Deleting a Data Source........................................................................................................................... 19 5.3.3 Creating Base Views from a JDBC Data Source .................................................................................... 19 5.3.4 Viewing the Schema of a Base View...................................................................................................... 22 5.3.5 Importing Web-Service-Type Sources .................................................................................................... 25 5.3.6 Creating Base Views from a Web Service.............................................................................................. 27

5.4 VISUAL CREATION OF VIEWS AND QUERIES................................................................................. 29 5.4.1 Creating Union-Type Views .................................................................................................................... 29 5.4.2 Creating Join-Type Views....................................................................................................................... 31 5.4.3 Creating Projection / Selection-Type Views........................................................................................... 35 5.4.4 Creating Aggregation-type Views........................................................................................................... 39 5.4.5 Flattening of Register-array-type Fields ................................................................................................. 41 5.4.6 Viewing the Schema of a View............................................................................................................... 46 5.4.7 Tree View Mode...................................................................................................................................... 46 5.4.8 Data Lineage ........................................................................................................................................... 47


5.4.9 Executing Views...................................................................................................................................... 48 5.4.10 Using VQL Shell....................................................................................................................................... 53 5.4.11 Editing / Replacement of a View Definition ........................................................................................... 54 5.4.12 Source Refresh........................................................................................................................................ 55 5.4.13 Views Affected by Modifications ........................................................................................................... 55

5.5 OTHER TYPES OF SOURCES............................................................................................................... 56 5.5.1 Path Types in Virtual DataPort ................................................................................................................ 56 5.5.2 ODBC Sources ......................................................................................................................................... 60 5.5.3 Multidimensional Database Sources...................................................................................................... 61 5.5.4 XML Sources ........................................................................................................................................... 63 5.5.5 JSON Data Sources ................................................................................................................................ 66 5.5.6 Web Sources (WWW)............................................................................................................................. 67 5.5.7 Delimited File Sources ............................................................................................................................ 68 5.5.8 Google Enterprise Search / Mini Sources .............................................................................................. 73 5.5.9 Aracne Sources ....................................................................................................................................... 77 5.5.10 LDAP Sources.......................................................................................................................................... 82 5.5.11 BAPI Sources........................................................................................................................................... 86 5.5.12 CUSTOM Sources.................................................................................................................................... 88 5.5.13 JDBC/ODBC sources through SQL Query................................................................................................ 92 5.5.14 Data Source Configuration Properties .................................................................................................... 96

5.6 ADVANCED CONFIGURATION OF VIEWS........................................................................................ 97 5.6.1 Internationalization Configuration .......................................................................................................... 98 5.6.2 Configuring the Cache of a View ............................................................................................................ 98 5.6.3 Swapping Configuration ......................................................................................................................... 99 5.6.4 Editing the Execution Plan .................................................................................................................... 100 5.6.5 Query Capabilities................................................................................................................................. 101 5.6.6 View Configuration Properties.............................................................................................................. 104 5.6.7 Configuration Properties for Specific View Types................................................................................ 105

5.7 STORED PROCEDURES ..................................................................................................................... 106 5.7.1 Importing Stored Procedures ................................................................................................................ 106 5.7.2 Executing Stored Procedures ................................................................................................................ 108 5.7.3 Use of Stored Procedures in Creating Views........................................................................................ 110

6 PUBLICATION OF WEB SERVICES.................................................................................................. 112 6.1 WEB SERVICE TYPES........................................................................................................................ 112 6.2 PUBLISHING WEB SERVICES.......................................................................................................... 113

6.2.1 Operations for Updatable Views........................................................................................................... 116 6.2.2 Configuration of Connections in the Web Services Published............................................................. 117 6.2.3 SOAP XSLT Transformations................................................................................................................. 117 6.2.4 REST XSLT Transformations ................................................................................................................. 118 6.2.5 SOAP over JMS..................................................................................................................................... 118 6.2.6 Mappings in the Publication as RSS..................................................................................................... 119 6.2.7 Web Services Authentication ............................................................................................................... 120 6.2.8 Types Conversion Table for REST / SOAP Published Web Services .................................................... 120

6.3 INVOKING THE EXPORTED WEB SERVICES ................................................................................. 121 6.3.1 HTML Output Configuration.................................................................................................................. 123 6.3.2 JSON Output Configuration .................................................................................................................. 124

7 PUBLICATION OF VIEWS AS WIDGETS......................................................................................... 125 7.1 PUBLISH A VIEW AS A WIDGET ..................................................................................................... 125 7.2 AUXILIARY WEB SERVICES ............................................................................................................. 127 7.3 EXPORT TO JSR-168 OR JSR-286 PORTLET................................................................................... 127 7.4 EXPORT TO OPENAJAX WIDGET.................................................................................................... 130 7.5 DEPLOYMENT OF AN OPENAJAX WIDGET .................................................................................. 131


7.6 EXPORT AS MICROSOFT WEB PART ............................................................................................. 138 7.7 DEPLOYMENT OF A MICROSOFT WEB PART............................................................................... 139

8 SERVER ADMINISTRATION ............................................................................................................. 141 8.1 CONFIGURING THE SERVER ............................................................................................................ 141

8.1.1 Server Connectivity ............................................................................................................................... 141 8.1.2 Threads Pool.......................................................................................................................................... 142 8.1.3 Configuring the Cache........................................................................................................................... 143 8.1.4 Limiting the Number of Concurrent Requests ...................................................................................... 145 8.1.5 Default Configuration of HTTP Proxy .................................................................................................... 145 8.1.6 Configuring the Wwapping to Disk Parameters ................................................................................... 146 8.1.7 Configuring Runtime Parameters for Stored Procedures...................................................................... 147 8.1.8 Configuring Default Internationalization .............................................................................................. 148 8.1.9 JMS Listeners ....................................................................................................................................... 149

8.2 EXPORTING / IMPORTING THE SERVER METADATA ................................................................. 152 8.3 IMPORTING EXTENSIONS ............................................................................................................... 155

9 DATABASES, USERS AND ACCESS RIGHTS IN VIRTUAL DATAPORT..................................... 156 9.1 DATABASES IN VIRTUAL DATAPORT............................................................................................ 156 9.2 USER AND ACCESS RIGHTS STRUCTURE IN VIRTUAL DATAPORT......................................... 156

9.2.1 User Types............................................................................................................................................. 156 9.2.2 Types of Access Rights ......................................................................................................................... 156

9.3 GRAPHIC ADMINISTRATION OF DATABASES, USERS AND ACCESS RIGHTS ..................... 158 9.3.1 Creating Databases............................................................................................................................... 158 9.3.2 Configuration and Deletion of Databases ............................................................................................ 158 9.3.3 Creating Users....................................................................................................................................... 160 9.3.4 Modifying and Deleting Users .............................................................................................................. 162

10 MONITORING THE VIRTUAL DATAPORT SERVER ....................................................................... 166 10.1 MONITORING WITH A JAVA MANAGEMENT EXTENSIONS (JMX) AGENT .......................... 166

10.1.1 Using JConsole ..................................................................................................................................... 167 10.1.2 General Information on the Server ....................................................................................................... 171 10.1.3 General Information on Data Sources .................................................................................................. 172 10.1.4 Information on the Cache...................................................................................................................... 173 10.1.5 Information and Events on Catalog Access (DDL Statements)............................................................. 173 10.1.6 Information and Events on the Running of DML Statements............................................................... 175 10.1.7 Information and Events on Transactions .............................................................................................. 176

10.2 DENODO MONITOR ........................................................................................................................... 177 10.2.1 Local Monitors ...................................................................................................................................... 178 10.2.2 Server Monitors .................................................................................................................................... 178 10.2.3 Virtual DataPort Queries Monitor ......................................................................................................... 179 10.2.4 Configuring the Denodo Monitor .......................................................................................................... 179

11 CLUSTER ARCHITECTURES / SERVER BACKUP ........................................................................... 181 11.1 TOOLS INSTALLATION...................................................................................................................... 181 11.2 USE OF THE PING SCRIPT ................................................................................................................ 181 11.3 USE OF THE IMPORT/EXPORT SCRIPTS FOR BACKUP AND/OR REPLICATION..................... 182 11.4 CONFIGURING SEVERAL INSTANCES OF A VIRTUAL DATAPORT SERVER............................ 184

11.4.1 How to Configure the Slave instances ................................................................................................. 184

12 APPENDICES ...................................................................................................................................... 186 12.1 JDBC DRIVERS ................................................................................................................................... 186


12.2 MULTIDIMENSIONAL TO RELATIONAL MAPPING ..................................................................... 188 12.3 USING THE ADVANCED CONDITIONS EDITOR ............................................................................ 190

12.3.1 Creating Values for the Conditions....................................................................................................... 191 12.3.2 Creating Simple Conditions .................................................................................................................. 192 12.3.3 Creating Boolean Conditions ................................................................................................................ 193 12.3.4 Creating Conditions that Use Compound-type Constants .................................................................... 193

12.4 CONFIGURING LOGS ......................................................................................................................... 198 12.5 INSTALLING THE OPENAJAX MASHUP EDITOR ......................................................................... 199 12.6 INSTALLING THE DENODO SOLUTION FOR MICROSOFT SHAREPOINT................................. 201 12.7 TRANSFORMING INCOMING/OUTGOING SOAP MESSAGES WITH XSLT STYLESHEETS... 202 12.8 JMS CONNECTION DETAILS: JNDI PROPERTIES AND CLIENT JARS..................................... 204

12.8.1 Apache ActiveMQ ................................................................................................................................. 204 12.8.2 IBM WebSphere MQ............................................................................................................................. 204 12.8.3 OpenJMS .............................................................................................................................................. 205 12.8.4 Progress SonicMQ................................................................................................................................. 205

12.9 INSTALLING THE CONNECTORS FOR SAP ERP, SAP BW AND SAP BI................................... 207 12.9.1 Installing the Connector for SAP ERP (BAPI Data Sources).................................................................. 207 12.9.2 Installing the Connector for SAP BW and SAP BI (Multidimensional Sources) ................................... 208

REFERENCES ................................................................................................................................................................ 209


FIGURES Figure 1 Virtual DataPort Architecture.................................................................................................................... 3 Figure 2 Authentication screen in DataPort Administration Tool........................................................................... 9 Figure 3 Creating projects..................................................................................................................................... 15 Figure 4 Importing a JDBC Data Source ............................................................................................................... 17 Figure 5 Configuring the JDBC connections pool ................................................................................................. 18 Figure 6 JDBC Data source configuration dialog.................................................................................................. 19 Figure 7 Schemas, tables and views of a JDBC source ....................................................................................... 20 Figure 8 Renaming new views with the same name as existing ones ................................................................ 21 Figure 9 Accepting the schema of a base relation ............................................................................................... 22 Figure 10 Schema of the base view phone_inc...................................................................................................... 23 Figure 11 Schema of the base view internet_inc................................................................................................... 24 Figure 12 Creating a Web service data source....................................................................................................... 27 Figure 13 Operations and parameters of a Web Service ....................................................................................... 28 Figure 14 Schema of the base view average_monthly_sales................................................................................ 29 Figure 15 Constructing the union view of phone_inc and Internet_inc ................................................................. 30 Figure 16 Constructing the ‘incidences’ and ‘average_monthly_sales’ join view................................................. 32 Figure 17 Schema of the incidences_sales join view ............................................................................. 34 Figure 18 Constructing the selection view pref_clients_inc_sales ................................................. 37 Figure 19 Creating a value of type getSumRevenuebyTaxIds_IN0 ..................................................... 39 Figure 20 Creating aggregation view inc_grouped_by_pref_clients............................................ 40 Figure 21 The operation getAverageMonthlyRevenue from the Web Service sales ..................... 42 Figure 22 Default base view created for the Web Service operation getAverageMonthlyRevenue .. 43 Figure 23 Flattening the array of the base view average_revenue_array ........................................... 44 Figure 24 Output schema of the base view average_revenue_array flattened................................... 44 Figure 25 Schema of the flat_revenue view............................................................................................... 45 Figure 26 Schema of the view incidences .................................................................................................... 46 Figure 27 Tree view of incidences_sales ................................................................................................ 47 Figure 28 Data lineage of incidences_sales ........................................................................................... 48 Figure 29 Executing the view incidences_sales..................................................................................................... 49 Figure 30 Result comprised of one tuple with a field of the type array of registers ............................................. 50 Figure 31 Content of a field of the type array of registers ..................................................................................... 51 Figure 32 Incidences view execution trace............................................................................................................. 52 Figure 33 Join view execution trace with errors .................................................................................................... 53 Figure 34 Confirmation of the replacement of a view............................................................................................ 55 Figure 35 View affected by modification to another view ..................................................................................... 56 Figure 36 Http POST request with XML Body......................................................................................................... 57 Figure 37 Providing input values to interpolation variables ................................................................................... 59 Figure 38 Importing from an ODBC datasource ...................................................................................................... 61 Figure 39 Creating a data source to a multidimensional database........................................................................ 62 Figure 40 Create a base view from an MDX query (multidimensional data source).............................................. 63 Figure 41 Importing from an XML data source ....................................................................................................... 64 Figure 42 RSS Xpath ............................................................................................................................................... 65 Figure 43 Description of an XML source................................................................................................................. 66 Figure 44 Importing from a JSON datasource ........................................................................................................ 67 Figure 45 Importing a WWW wrapper ................................................................................................................... 68 Figure 46 Importing from a new delimited file source ........................................................................................... 69 Figure 47 Example 1 of tuple pattern: contents of the delimited file..................................................................... 70 Figure 48 Example 1 of tuple pattern: regular expression to extract the contents of the file ............................... 70 Figure 49 Example 2 of tuple pattern: output of the Windows command DIR....................................................... 70 Figure 50 Example 2 of tuple pattern: regular expression to extract the output of the command DIR ............... 70 Figure 51 Description of a delimited-file-type source ............................................................................................ 72 Figure 52 Importing a new source of type Google Mini ......................................................................................... 74 Figure 53 Description of the Google Mini source................................................................................................... 75


Figure 54 Base View Creation in Google Enterprise Search / Mini Datasources .................................................. 76 Figure 55 Creating an Aracne data source ............................................................................................................. 78 Figure 56 Description of the Aracne source ........................................................................................................... 79 Figure 57 Creating base views in Aracne sources.................................................................................................. 80 Figure 58 Adding an attribute with the most relevant terms of the SEARCHABLECONTENT field....................... 81 Figure 59 Creating an LDAP data source ................................................................................................................ 83 Figure 60 New LDAP data source ........................................................................................................................... 84 Figure 61 New LDAP base view from expression................................................................................................... 85 Figure 62 Accessible objects of an LDAP base view created from an expression................................................. 86 Figure 63 Creating a new BAPI data source ........................................................................................................... 87 Figure 64 Creating a new BAPI (SAP ERP) data source .......................................................................................... 88 Figure 65 Importing a CUSTOM wrapper................................................................................................................ 89 Figure 66 New CUSTOM data source..................................................................................................................... 90 Figure 67 Creating a base view from a CUSTOM data source............................................................................... 91 Figure 68 New base view from a CUSTOM data source........................................................................................ 92 Figure 69 Creating a base view using an SQL query .............................................................................................. 93 Figure 70 Editing the value of the interpolation variables ..................................................................................... 94 Figure 71 Specifying aliases ................................................................................................................................... 95 Figure 72 Advanced Configuration screen of the view average_monthly_sales ................................. 98 Figure 73 Cache invalidation of a view................................................................................................................... 99 Figure 74 Editing the Execution Plan of a view .................................................................................................... 101 Figure 75 Search form for a bookshop.................................................................................................................. 103 Figure 76 Search method for a bookshop ............................................................................................................. 103 Figure 77 Importing a Stored Procedure ............................................................................................................... 107 Figure 78 Properties of a Stored Procedure.......................................................................................................... 108 Figure 79 Execution of the Stored Procedure CalculateAvgRevenue ................................................... 109 Figure 80 Result of the execution of the stored procedure CalculateAvgRevenue.............................. 110 Figure 81 Parameters of a procedure added to a new view................................................................................. 111 Figure 82 Publishing a Web Service ..................................................................................................................... 112 Figure 83 Web services......................................................................................................................................... 114 Figure 84 Operations created for the view inc_grouped_by_pref_clients ................................. 115 Figure 85 Widgets Status Table ........................................................................................................................... 126 Figure 86 JSR286 Portlet ...................................................................................................................................... 129 Figure 87 Portlet search form................................................................................................................................ 130 Figure 88 Adding a new item to the OpenAjax Repository .................................................................................. 132 Figure 89 Adding the new widget to the OpenAjax Mashup Editor..................................................................... 133 Figure 90 OpenAjax Widget displaying the content of the ‘incidents_sales’ view. ............................................ 134 Figure 91 Opening the OpenAjax widget configuration form............................................................................... 135 Figure 92 OpenAjax widget search form .............................................................................................................. 136 Figure 93 ‘Edit Widget Properties’ dialog: binding properties of a widget with properties of other widgets .... 137 Figure 94 Binding the property of a widget with the property of another widget. .............................................. 138 Figure 95 Loading the “.webpart” file into the Web Part gallery......................................................................... 139 Figure 96 Web Part search form ........................................................................................................................... 140 Figure 97 Port Configuration ................................................................................................................................. 142 Figure 98 Cache configuration .............................................................................................................................. 144 Figure 99 Concurrent Requests Configuration...................................................................................................... 145 Figure 100 HTTP Proxy Configuration ..................................................................................................................... 146 Figure 101 Swapping Configuration ....................................................................................................................... 147 Figure 102 Configuration of Stored Procedures runtime parameters..................................................................... 148 Figure 103 Response message sent by a JMS listener.......................................................................................... 149 Figure 104 Response message to a DML query ..................................................................................................... 149 Figure 105 Creating a new JMS listener................................................................................................................ 151 Figure 106 List of existing JMS listeners ............................................................................................................... 152 Figure 107 Export dialog ......................................................................................................................................... 153 Figure 108 Creating a new database...................................................................................................................... 158 Figure 109 Description of a database..................................................................................................................... 159 Figure 110 Assigning privileges on a database...................................................................................................... 160


Figure 111 Creating a normal-type user ................................................................................................................. 161 Figure 112 Creating an LDAP-type user.................................................................................................................. 162 Figure 113 Description of a user............................................................................................................................. 163 Figure 114 Assigning privileges to a user at database level.................................................................................. 164 Figure 115 Assigning privileges to a user at view and/or stored procedure level................................................. 165 Figure 116 Connection to DataPort from jConsole ................................................................................................. 167 Figure 117 Virtual DataPort MBeans ...................................................................................................................... 168 Figure 118 Attributes tab of the RequestsManagementInfo Mbean ..................................................................... 169 Figure 119 Details of a query in jConsole ............................................................................................................... 170 Figure 120 Details of a notification in jConsole ..................................................................................................... 171 Figure 121 Relational star schema ......................................................................................................................... 188 Figure 122 MDX query for Visual result mapping (1).............................................................................................. 188 Figure 123 Editor of selection conditions ............................................................................................................... 190 Figure 124 Creating a constant value ..................................................................................................................... 192 Figure 125 Creating a simple condition .................................................................................................................. 193 Figure 126 RevenueSum schema ...................................................................................................................... 194 Figure 127 Compound values editor ....................................................................................................................... 195 Figure 128 Creating a value of type getSumRevenuebyTaxIds_IN0 ................................................... 196 Figure 129 Condition using compound-type values................................................................................................ 197 Figure 130 Result of executing the view RevenueSum .................................................................................... 197 Figure 131 SAP Java Connector (JCo) test screen ................................................................................................. 208


TABLES Table 1 Conversions between DataPort data types and Web service parameter types................................... 121 Table 2 URLs of Web Service’s information pages ........................................................................................... 121 Table 3 Obtaining the XSD Schema of a REST operation ................................................................................. 122 Table 4 Invoking a Web Service without parameters ....................................................................................... 122 Table 5 Invoking a Web Service with one parameter ....................................................................................... 122 Table 6 Invoking a REST Web Service with an array-type parameter .............................................................. 123 Table 7 Invoking the HTML Web Service with configuration parameters ........................................................ 124 Table 8 Invoking the JSON Web Service with padding (JSONP) ...................................................................... 124 Table 9 JDBC Drivers tested with Virtual DataPort........................................................................................... 186 Table 10 Contents of a SAP BW base view with Visual result mapping (1) ....................................................... 189 Table 11 Default values for Apache Active MQ .................................................................................................. 204 Table 12 Default values for IBM WebSphere MQ............................................................................................... 205 Table 13 Default JNDI properties for OpenJMS ................................................................................................. 205 Table 14 Default JNDI properties for Progress SonicMQ ................................................................................... 206


Preface i

PREFACE

SCOPE

Denodo Virtual DataPort provides business applications with easy access to integrated views of various heterogeneous, distributed and structured and semi-structured data sources. By using examples, this document introduces the reader to the installation, configuration and administration of Denodo Virtual DataPort, including how to create unified views of heterogeneous and distributed data sources.

WHO SHOULD USE THIS DOCUMENT

This document is aimed at developers and administrators that require a detailed knowledge of how to install, configure and administrate Virtual DataPort, including the creation of unified data views on heterogeneous, distributed, and structured and semi-structured data sources. The detailed information required to develop client applications that access Virtual DataPort is provided in the Developer Guide.

SUMMARY OF CONTENTS

More specifically, this document:

• Presents the fundamental concepts of the data integration solutions based on Virtual DataPort.

• Describes the processes required to install and configure Virtual DataPort.

• Gives a detailed description, through examples, of how the administration tools of Virtual DataPort are used to construct unified views of data from distributed and heterogeneous sources.


Introduction 1

1 INTRODUCTION

Nowadays, any organization of a certain importance makes use of a multitude of data sources developed over a period of time. Generally, these data sources are developed using different technologies (relational databases, Web services, XML documents, spreadsheets, flat files, etc.), using very heterogeneous data models, and include both structured and non-structured data. Development of new services for clients and/or optimization of company business processes require the new systems built to integrate the data stored in these inherited data repositories. In addition, it is often needed to access external data sources to interact with partners, providers or customers. The Web itself is also a very valuable information source for many business purposes. Virtual DataPort is a global solution for the real-time integration of heterogeneous, distributed, structured and semi-structured data sources. For this, it combines various features:

• Integrates and manages the data that are relevant to the company, regardless of its origin and format.

• It incorporates these data into its data system, in real time or with configurable preloads.

• Facilitates the construction of distributed information services with high strategic and functional value.

This document provides a technical guide to the use of Virtual DataPort.


General Architecture 2

2 GENERAL ARCHITECTURE

In outline, Virtual DataPort enables business applications to process a series of distributed and heterogeneous data sources, including external sources, as though the data were contained in a large ”Virtual” Database. This “virtual database” allows creating views that combine and integrate said data. Virtual DataPort acts as a mediator that provides a structured and unified view of the data contained in all the data sources included in the system. The system can easily deal with a wide range of structured, semi-structured and unstructured data sources such as: databases, Web sites, spreadsheets, XML documents, Web Services, LDAP servers, flat text files, indices on unstructured information, etc. The system allows easily importing each data source into the “virtual database”. Then, a SQL-like language called Denodo VQL (Virtual Query Language) is used to create views that arbitrarily combine the data of said relations using operations such as selections, projections, unions, joins, groups, etc, thus creating unified views of the source data. In this process, Virtual DataPort is able to work with sources that have limited query capacity (for example, in many Web services only queries specifying certain mandatory parameters are allowed). Furthermore, the Virtual DataPort query engine is capable of querying non-structured data and combining it with structured and semi-structured data. When the system receives a VQL query on a previously defined view, it can generate an execution plan for the query, which consists of a list of subqueries that are sent in real-time to the various sources involved and a series of operations combining the data obtained from each source. Furthermore, Virtual DataPort incorporates a system (called cache module) which allows the administrator to decide on the mechanism to be used for accessing the source data:

• The system can access the source data in real time, thus providing totally updated data.

• Caches can be created and configured for the sources or views as required.

Virtual DataPort also allows the updating of data sources, provided that these are capable of supporting transactions. The data integration system provided by Virtual DataPort is modeled on three separate layers: the user layer, the logical layer and the physical layer (wrappers). Figure 1 provides a general overview of the system architecture. Each of these layers is described below.



Figure 1 Virtual DataPort Architecture

2.1 PHYSICAL LAYER

The physical layer abstracts the higher architecture layers of the difficulties of interacting with the data sources. It also provides a vision of the data sources according to a common metamodel. These tasks are carried out through the so-called wrappers. A wrapper extracts data from a source, interprets the results obtained and returns them to the system in the format required by Virtual DataPort metamodel. Furthermore, where permitted by the source, a wrapper can also insert, update and/or delete information in a source. The wrappers allow Virtual DataPort to process all the external sources in a consistent manner, without concern for their specific characteristics. Virtual DataPort directly supports the following types of wrappers:

• Relational databases: They extract data from a Remote Database via JDBC or ODBC. Furthermore, where permitted by the source, a wrapper inserts, updates and/or deletes information in a source.

• Multidimensional databases: They extract data from multidimensional databases such as SAP BW and SAP BI.



• Web Services: They extract data invoking operations provided by SOAP Web services.

• XML: They allow extracting data encapsulated in XML files. Those files can optionally be validated using a specific DTD or schema. XML documents can be accessed on the local drive or remotely accessed via protocols such as http or FTP. A common use of this type of wrapper is the importing of REST-type Web services (including RSS and ATOM web services).

• JSON: Extract data from documents in JSON format [JSON]. They are useful to access REST-type Web services returning their output in JSON format.

• Flat files: Extract data from flat files in CSV format (Comma Separated Values) or similar. CSV documents can be accessed on the local drive or on remote locations via http or FTP.

• Web: Provide access to semi-structured data contained in websites. For instance, these wrappers can automate the process of querying a web form and extracting the results. These wrappers are generated with Denodo ITPilot [ITPILOT].

• Aracne: Denodo Aracne [ARCN] allows for the crawling, indexing and search for non-structured data contained in repositories such as Websites, relational databases, local file systems and e-mail servers. Indexes created using Denodo Aracne may be imported directly to Virtual DataPort for querying and combining with structured and semi-structured data.

• Google Enterprise Search / Google Mini: Google Enterprise / Google Mini [GMINI] is Google’s corporate solution for crawling, indexing and searching websites. The indexes created using Google Mini may be imported directly into Denodo Virtual DataPort for querying and combining them with structured and semi-structured data.

• LDAP: Extracts data contained in LDAP directories such as Microsoft Windows Server Active Directory [MS-AD].

• BAPI: They invoke SAP BAPIs (Business Application Programming Interfaces) to extract data stored in SAP ERP and other SAP applications.

• CUSTOM (also named as MY wrappers): They extract data from a source through a Java implementation provided by the Virtual DataPort administrator. Virtual DataPort provides an API, so users can create their own wrappers for specific sources. The CUSTOM wrappers also allow inserting, updating and/or deleting data from the sources.

Wrappers for databases, Web services, flat files, Aracne, Google Mini and XML are automatically created by Virtual DataPort as the corresponding data sources are imported (see 5.2). The wrappers for semi-structured Web sources can be created with Denodo ITPilot [ITPILOT].

2.2 LOGICAL LAYER

The logical layer integrates and combines the relations exported by the different wrappers (called base relations or base views) to create the views that will comprise the system global schema.



Once the base relations representing the system sources have been created, the administrator can create views that intercombine them as required, thus creating the global schema views (or derived views). It is important to point out that this process can be carried out in a recursive manner in several steps: a derived view can be used as a base to create new views, thus allowing combinations of arbitrary complexity. Views are defined using the Denodo VQL language [VQL] although, as explained in the next sections, the administration tool allows graphically creating them, so the VQL statements do not have to be manually written. Once the views of the global schema have been created by combining source data, the logical layer is capable of responding to queries expressed in VQL both on derived views and on base relations. The VQL query language is SQL-based, but it incorporates different extensions to handle heterogeneous and distributed data. For example, VQL includes certain commands to allow querying non-structured data and combining it with structured data. It also supports compound types such as arrays and registers. When the system receives a query, it checks that it can be resolved depending on the query capabilities supported by the data sources, it then draws up the possible execution plans, selects the most suitable one and executes the query returning the results obtained to the higher layer. The logical layer of Virtual DataPort also allows writing to data sources using INSERT/UPDATE/DELETE operations, provided that these are capable of supporting transactions. The following modules can be differentiated in the logical layer:

• Query Plan Generator: Firstly, the plan generator decides if the query received can or cannot be answered in accordance with the query capabilities supported by the data sources. Where it is possible, it generates the possible execution plans for the query.

• Optimizer: Aims to select the optimum execution plan from all the options (obtained by the Query Plan Generator). The query capabilities of the sources are also considered so, when possible, the processing of some operations is delegated to the data sources, thus achieving more efficient execution and less data exchange through the network. Other aspects taken into account are the most optimal execution strategies for join operations.

• Query Execution Engine: Once the optimum plan has been selected, the execution engine is responsible for putting it into practice, executing the necessary subqueries on the sources and integrating the results obtained to generate the global response. In turn, the execution engine takes into consideration that information from the sources which is already preloaded in the cache module, whereby unnecessary access to data sources is avoided, thus achieving greater efficiency.

2.2.1 Data Module: Cache

As mentioned above, Virtual DataPort has a system (called cache module) to store local copies of the source data as required. This cache is stored in a relational database accessible through JDBC. Virtual DataPort embeds an Apache Derby [DERBY] database that can be used to store the cache information. It is also possible to use the following external DBMSs to store the cache: MySQL [MYSQL], Microsoft SQL Server [MS_SQL]and Oracle [ORCL]. The system will generate and maintain a table in the cache database for each base view or view for which the cache has been enabled. As the queries are made, the tables are filled with the rows obtained from the sources. Each view in the cache has an associated expiration time, whereby the “expired” tuples are not considered in the queries and are automatically deleted at regular intervals. The cache can be preloaded periodically by writing a query that retrieves all the required data and scheduling this query to be repeated at the desired time interval.



The cache can also be disabled for a specific query or for the desired views, whereby it is always possible to access the data in the sources in real time. The section 8.1.3 explains how to enable the cache module.

2.3 USER LAYER

The user layer implements the interface between the client applications and the Virtual DataPort server. Thus, its responsibility is to provide client applications with means to establish a connection with the server, send queries and obtain the results. When the logical layer returns a response to the query, this layer is also responsible for providing it to the client application through the connection interfaces defined to this effect (see Virtual DataPort Developer Guide [DEV]). Denodo Virtual DataPort can be accessed via JDBC, ODBC, SOAP, REST, JSON and RSS Web Services (see section 6), JSR-168/286 Portlets, Open Ajax widgets and Microsoft Web Parts (see section 7), and through Virtual DataPort JAVA native API.


Installation and Initial Configuration 7

3 INSTALLATION AND INITIAL CONFIGURATION

The Denodo Platform Installation Guide document [DENINST] provides all the needed information to install Virtual DataPort, including the minimum hardware and software pre-requisites. It also includes instructions for using the installation tool and to perform the initial system configuration tasks.


Execution 8

4 EXECUTION

To start and stop the Virtual DataPort server, there are two options:

• The Denodo Platform Control Center (see Denodo Platform Installation Guide [DENINST]) allows, among other functionality, to start and stop all servers and tools of the Denodo Platform.

• Executing the script: $DENODO_HOME/bin/vqlserver.sh startup (vqlserver.bat startup in Windows) to start the server. To stop it, execute $DENODO_HOME/bin/vqlserver.sh shutdown (vqlserver.bat shutdown in Windows).

To start the Virtual DataPort administration tool, there are two options;

1. Using the shortcut included in the Denodo Platform Control Center [DENINST].

2. Using the script: $DENODO_HOME/bin/vdpadmin.sh (vdpadmin.bat or vdpadmin.exe in Windows).

4.1 EXECUTING THE VIRTUAL DATAPORT GRAPHIC ADMINISTRATION TOOL

Once the Virtual DataPort graphic administration tool has been launched, the first step is authenticating in the DataPort server (see Figure 2). The server uri format is: //name_machine:port/database (e.g. //localhost:9999/admin) where:

• name_machine: name of the machine in which the server resides. • port: number of the server execution port. • database: (optional) Virtual DataPort database to which you wish to connect. When a Virtual DataPort

server is installed, a default database called admin is created.


Execution 9

Figure 2 Authentication screen in DataPort Administration Tool

For more details on the concepts of databases and users in Virtual DataPort, see section 9. If this is the first access to the server, the default administrator user can be used (user login “admin” with password “admin”). Users should change their password after connecting to the server for the first time. For this, open the ‘Change password’ dialog on the ‘File’ menu. There are two modes for connecting to the server: database administration mode and server administration mode. The database administration mode is accessed when the server uri specifies a database and, in addition, the specified user is not an administrator. This mode allows managing the data sources, base relations and unified views that form part of a specific database of the DataPort server. The server administration mode is reserved for administrator users and, in that case, it is not mandatory that the supplied uri specifies a database. This mode allows executing tasks like those listed below:

• Create, modify or delete databases (see section 9). • Create, modify or delete users and manage their access privileges (see section 9). • Configure the cache system (see section 8.1.3). • Configure the swapping policy (see section 5.6.3). • Configure the server thread pool (see section 8.1.2). • Configure the server execution and stop ports (see section 8.1.1). • Configure the internationalization options (see section 8.1.8). • Configure other preferences such as HTTP proxy preferences (section 8.1.5), runtime parameters for stored

procedures (section 8.1.7) and limiting the number of concurrent requests processed by the server (section 8.1.4),

• Configure JMS listeners to receive queries from a JMS server (see section 8.1.9) If the specified URI includes a database and the user is an administrator, then the administration tool will allow her to manage both the server and the specified database.


Execution 10

4.2 DOCUMENTATION AND ONLINE HELP

Online help for the Virtual DataPort administration tool can be accessed by choosing the “Online Help” option on the “Help” menu. It is also possible to access contextual help information about the current dialog by clicking on the ‘?’ icon on the upper-right section of the screen. The online help is organized in topics according to the table of contents shown on the left side of the Help screen. It is also possible to perform text-based searches on the help contents by using the ‘Find’ tab. The product manuals are also available. They can be accessed selecting the option ‘Denodo Platform Documentation’ in the ‘Help’ menu (a new window browser will be opened)

4.3 MAIN AREAS OF THE ADMINISTRATION TOOL

The Virtual DataPort Administration tool has three main areas: the menu toolbar, the task pane and the workspace. The menu toolbar is placed on the top of the screen and includes the following options:

• The ‘File’ menu allows closing the session (‘Disconnect’ option), exiting the tool, changing the password of the active user, importing or exporting metadata for creating/restoring backups (see section 8.2), importing new extensions to the tool (see section 8.3), and refreshing the metadata from the catalog. Refreshing is useful when creating new elements in the catalog by directly using the VQL language, instead of creating them graphically.

• The ‘View’ menu allows configuring the administration tool preferences (see section 4.4) and managing the list of ‘hidden’ views. This list contains views that are not shown in the menus of the task pane on the left of the screen because they have been manually ‘hidden’ by the user (see section 5.2.1). Clicking on one of the hidden views will make it visible again. To hide a view, simply right-click on its name on the task pane and choose the ‘Hide view’ option.

• The ‘Tools’ menu allows selectively invalidating the server cache (see section 4.5) and searching the

DataPort metadata catalog (see section 4.6.2).

• The ‘Help’ menu allows accessing the online help and the online manuals (see section 4.2). The task pane is composed of a list of sections, each one containing a hierarchy of drop-down menus. The available sections are:

• Project Management. It allows creating and removing projects (see section 5.2).

• Connect. It allows importing data sources and creating base views (see sections 5.3 and 5.5). • Combine. It allows creating new views by combining other views through operations such as union,

join, projection/selection, etc. It also allows editing existing views, and examining the dependencies between them. See section 5.4.

• Publish. It allows publishing views and/or stored procedures as Web services (SOAP, REST, JSON,

RSS, …) and / or visual widgets (JSR-168/286, Microsoft Web Part, …). See section 7. • VQL Shell. It allows manually writing and executing VQL statements. See section 5.4.10.

For administrator users, the task pane also has the following sections:

• Database Management. It allows managing the databases of the DataPort server. See section 9.


Execution 11

• User Management. It allows managing users and access rights. See section 9.

• Server configuration. It allows configuring different aspects of the server. See section 8.

The menu tool bar also contains the ‘Quick Search’ drop down list (see section 4.6.1). This list is a shortcut for the menus in the task pane. For example, if the current section is Combine, the task pane shows the list of Base Views, Views and Stored Procedures in each project. Therefore, the Quick Search drop-down list will contain the same elements and selecting one of them will have exactly the same behavior than selecting it on the task pane.

4.4 TOOL PREFERENCES

The ‘View-Preferences’ menu of the graphical administration tool allows configuring certain preferences described in the following subsections.

4.4.1 Locale

The Locale configuration of the Administration Tool controls the appearance of data that depends on the user’s region. I.e. the formatting used to display the fields of type date. This can be changed in the dialog of the ‘View – Preferences – Locale’ menu. Note: This preference does not change the internationalization preferences of the Virtual DataPort server. See section 8.1.8 for information about changing this preference of the server.

4.4.2 Connection Parameters

The connection between the Administration Tool and the Virtual DataPort server can be configured in the ‘View – Preferences – Connection’ dialog. It is possible to configure the following parameters:

• Query Timeout: Maximum time (in milliseconds) the Administration Tool will wait for a query to finish. If 0, it will wait indefinitely until the sentence ends.

• Chunk Size: The results obtained by executing a sentence can be divided into blocks (chunks), so the

Server does not have to wait until a sentence ends, to send the processed tuples to the Tool. This parameter establishes the maximum number of results that a block can contain. When the Virtual DataPort server obtains enough results to complete a block, it sends this block to the client and continues processing the next results.

• Chunk Timeout: This parameter establishes the maximum time (in milliseconds) the Server waits before

returning a new block to the Tool. When this time is surpassed, the Server sends the current block even if it does not contain the number of results specified in the Chunk Size parameter. Note: if the values of Chunk Size and Chunk Timeout are 0, the Server returns all the results in a single block. If both values are different than 0, the Server returns a chunk whenever one of these conditions happen first:

o The chunk is filled (Chunk Size) o Or, after a certain time of not sending any chunk to the client (ChunkTime).

• Default Server URL: The default Server URL that appears in the authentication dialog of the

Administration Tool.


Execution 12

4.5 INVALIDATE CACHE

This option lets the user delete the content that is currently stored in the cache for one or more views. Open the ‘Invalidate cache’ dialog by clicking on the menu ‘Tools – Invalidate Cache’. Invalidate the cache content of one or more views by selecting them and clicking the Invalidate button. Check the Invalidate on cascade option to invalidate all the views underlying this current view. Example: if there is a view named V3, which is a projection of the base views BV1, and BV2 and you Invalidate the cache on cascade of the view V3, automatically the cached content of the views BV1 and BV2 will be also automatically invalidated.

4.6 SEARCH

A user can search an element of the Virtual DataPort catalog using:

• Quick Search: to easily search by name. • Catalog Search: to search objects in the catalog using more advanced filters such as Element Type,

Creation Date, Description, etc.

4.6.1 Quick Search

The easiest way to open an element is searching its name in the Quick Search drop-down list located at the top of the Administration Tool and then clicking on its name. The elements accessible from the Quick Search drop-down list are exactly the same as the ones shown in the pane on the left side of the tool. Depending on the current section, the elements shown may vary. For example, if the current section is Combine, the panel on the left contains a list of Base Views, Views and Stored Procedures. Therefore, the Quick Search drop-down list will contain the same elements.

4.6.2 Catalog Search

The Catalog Search tool searches all the elements in the catalog of the current Virtual DataPort server database. It is more powerful than Quick Search because it allows the user to search, not only by name, but also by type, owner, description, etc. Catalog Search can be accessed by selecting the option “Catalog Search” in the “Tools” menu. The possible criteria for searching elements are the following:

• Element Type. If the selected element type is ‘Views’, the user can also select ‘View Type’, ‘Swap active’ and ‘Cache Status’ as search criteria.

• Owner. User name of the user that created the elements.

• Last User Modifier. User name of the last user that modified the elements.

• Initial Creation Date and End Creation Date. Elements that were created between the ‘Initial Creation

Date’ and the ‘End Creation Date’.

• Initial Modification Date and End Modification Date. Elements that were modified between the ‘Initial Modification Date’ and the ‘End Modification Date’.


Execution 13

• Description. Elements that contain the value of this field in its description. It is possible to open the configuration of an element included in the results by clicking on its name.

4.7 QUERY MONITOR

The Query Monitor lists the queries that the Virtual DataPort server is currently executing. If we select a row of the table, the Tool displays the Tree View of the query. That is, the tree of views used to form the results of the query. Click on each node of the tree to display information about it, such as number of processed rows, state, etc. You can copy the value of each parameter by right-clicking on it and then, clicking on ‘Copy’. To store the current tree of views into an image file, click on ‘Save image’. The Monitor updates automatically the list of queries unless we deselect the option ‘Automatically refresh every…’ In this case, we can click on ‘Update’ to update the list. To stop the execution of a query, select its row in the table and click on ‘Stop query’.


Visual Creation of Views 14

5 VISUAL CREATION OF VIEWS

This section describes how to use the Virtual DataPort graphic administration tool to create unified data views graphically from various heterogeneous sources. We will use the following example as a guide when describing the process: Example: Unified data about customer sales and incidents. A telecommunications company offers phone and internet services to its clients. Data on the incidents reported in the phone service are stored in a relational database, which is accessed through JDBC. And data on the incidents reported in the Internet service are stored in another relational database also accessed through JDBC. In our example, the director of the I.T department wants to monitor the number of incidents (either telephony or Internet) notified by the clients with the greatest sales volume to establish whether or not measures should be taken to increase client satisfaction. Data on customer sales volumes are managed by another department of the company. That department provides a Web Service so the other departments can access to that data. In this example, we will see how Virtual DataPort can be used to construct a unified data view to meet the needs of the I.T department, by obtaining the total number of incidents from clients with the greatest sales volumes. The path DENODO_HOME/samples/vdp/incidences contains:

• SQL scripts for creating the tables used in the examples of this manual (version for MySQL, Oracle and PostgreSQL). Note: to follow the examples of this guide, use one of these scripts to create the required tables in a database.

• A .war file with the implementation of the Web Service used in the examples. It has been tested with Tomcat 5.x [TOM] and Axis 1.x [AXIS].

• A WSDL file with the description of the Web service used. • vql scripts (VQL stands for Virtual Query Language [VQL]) that create the objects (data sources, views,

stored procedures…) that we will learn to create in this manual. You don’t need to use them if you are going to follow this guide. Otherwise, edit the vql script that matches your database:

• Replace ‘@HOSTNAME’ with the host name of your database. • In the ‘CREATE WRAPPER JDBC’ commands, change the parameter ‘RELATIONNAME’,

so it matches the name of the schema in your database. I.e: change RELATIONNAME='VDB.INTERNET_INC' for RELATIONNAME=’test.INTERNET_INC'.

5.1 ACCESS TO THE GRAPHIC ADMINISTRATION TOOL

The first step is to access the tool in database administration mode (see section 4.1). If a database and users have not been created yet (see section 9), you can use the username (admin) to connect to the default database (vdp).



5.2 CREATING AND MANAGING PROJECTS

The different components (data sources, base relations, views, stored procedures, web services and widgets) that can be created using the Virtual DataPort administration tool are grouped together into projects. On starting the tool for the first time, there will be only one project available called ‘Default’. To create a new project, select the ‘Project Management’ tab and click on the option ‘new’ (see Figure 3). On this screen, give a name to the project and, optionally, a description. The new project will be created on clicking ‘ok’. In our example, the name of the new project is ‘VDPExample’.

Figure 3 Creating projects

A project can be edited by clicking on its name in the drop-down menu on the left of the screen or deleted by clicking on the ‘drop’ option. If a project is deleted, the elements it contains are automatically moved to the ‘Default’ project. NOTE: these projects only exist in the graphical administration tool used to create them.

5.2.1 Moving Elements to Other Projects

As we will see in the next sections, on creating a new datasource or a new view, the new element can be allocated to one of the existing projects. To move an existing element to another project, proceed as follows:

1. Expand the menus on the left of the screen until the required element has been located. The data sources will be accessible from the ‘Connect’ tab and the views, from the ‘Combine’ tab.

2. Right-click on the element to be moved and use the ‘Move to Project’ option to select the destination

project.



5.3 IMPORTING DATA SOURCES AND CREATING BASE VIEWS

The first step in our example is to import the data sources and create base views (also called base relations) which represent the data necessary to construct the unified views in Virtual DataPort. In the following sections, we will deal with:

• Importing ‘Database’ sources and creating base views from their tables, which will allow us to import the databases of incidents of our example (sections 5.3.1 and 5.3.3).

• Importing Web-Service-type sources and creating base views from their operations (sections 5.3.5 and

5.3.6), that will be used to import data on sales.

• Deleting data sources (section 5.3.2).

• Examining the schema of views (section 5.3.4). How to import other types of data sources and how to create base views from them is explained in section 5.5.

5.3.1 Importing JDBC-Type Sources

To create a new DataSource, open the “Connect” tab on the left side of the tool. Then, click on the project to which the new source is to be added. In our example, select the ‘VDPExample’ project. When the ‘Relational DB’ menu is dropped down, it displays the list of external databases added as sources to the Virtual DataPort database. They appear classified in two categories: JCBC and ODBC. To add a new JDBC database to the list, click on the ‘New’ button beside the JDBC database list. You then access the screen for creating a new JDBC data source (see Figure 4). The following data are then requested in this dialog:

• Name. Name of the new data source in Virtual DataPort. In our example, we will use the name phone_ds for the data source of the telephone service table. And internet_ds for the data source of the Internet service incidents table.

• Database Adapter. In this selectable it is possible to choose one of the DBMSs that Virtual DataPort

includes a specific adapter for. A “Generic” adapter is also included for using with JDBC-accessible DBMSs without a specific adapter. Each specific adapter implements several optimizations for its target DBMS, so they should be used whenever possible. If the Choose automatically option is selected (recommended), DataPort will discover the most suitable adapter to connect to the DBMS. On selecting an adapter, the Driver Class field is completed automatically and the “DB Uri” field is completed with a “template” of the uri needed to connect to the selected database. The Appendix 12.1 contains a list of the JDBC drivers included and/or normally used.

• Driver JAR file. Path to the .jar file containing the JDBC driver to be used (optional). When using a

JDBC driver not included with Virtual DataPort, click ‘Browse’ to select the .jar file containing the required driver. Note: This is only necessary if the driver is not included with Virtual DataPort and the driver has not been included in the CLASSPATH of the server. To include a new driver in the classpath, place its .jar file in the DENODO_HOME/extensions/thirdparty/lib path (DENODO_HOME is the Denodo Platform installation directory) and restart the server.

• Driver class. Name of the JAVA class of the JDBC driver to be used. For example, for a MySQL database

the value com.mysql.jdbc.Driver is normally used. The appendix 12.1 contains a list of the values that must be used with the JDBC drivers that are included and/or most commonly used with Virtual



DataPort.

• DB URI. Access uri to the database. In our example, we assume that both tables of incidents are stored in the database incidents_center of a MySQL server located in the machine ‘acme’. So, the URI is: jdbc:mysql://acme/incidents_center. The appendix 12.1 contains a list of the values that must be used with the JDBC drivers that are included and/or most commonly used with DataPort.

• Login. User login to access the external database.

• Password. User password to access the external database.

• If the Choose automatically option is checked (recommended), the system will connect to the DBMS to

discover the most suitable adapter. On marking this option the ‘Test Connection’ option is automatically selected.

• Finally, if the Test Connection option is selected, the system will connect to the source to verify it can be

accessed from the DataPort server.

Figure 4 Importing a JDBC Data Source

Optionally, we can add properties that will be used as arguments when establishing a connection with the database. To do this, click on ‘Driver properties’ Note: if this data source will connect to Oracle and we want the data source to list its synonyms, we have to add the property ‘includeSynonyms’ = ‘true’. Open the ’Connections Pool Configuration’ dialog to configure several parameters of the connections pool that Virtual DataPort will use to optimize access to the external database (see Figure 5):



• Initial Size: number of connections with which the pool is to be initialized. A number of connections are established and created in “idle” state, ready to be used.

• Max Active: maximum number of active connections the pool can manage at the same time (a negative

value implies there is no limit).

• Ping Query: SQL query used by the pool to verify the status of the connections. The query should consume as little resources in the database as possible, and the table in question should exist.

• Test Connection: if this property is checked and there is a specified ping query, each connection retrieved

from the connection pool will be validated by executing the ping query. Note: in production environments, we strongly recommend selecting the Test connection checkbox and defining a Ping query.

Figure 5 Configuring the JDBC connections pool

After creating the data source, we can go back to its configuration dialog by clicking on the data source in the left-side menu and then on the ‘Edit’ button. In the ‘Edit JDBC Data Source’, we can configure the specific behavior of the JDBC data source such as:

• Which operations are delegated to the database (executed in the database instead of in Virtual DataPort Server)

• Which operations this data source has support for. See section 5.5.14 for more details about this dialog. Note: In most cases, the default value for these options is correct. Therefore, this dialog will only be useful in very particular environments.



Figure 6 JDBC Data source configuration dialog

5.3.2 Deleting a Data Source

Once a data source has been added, it can be deleted it by clicking on the ‘drop’ button beside the data source name in the ‘Connect’ tab. If there are other elements in the DataPort catalog (e.g. views) which depend on that data source, the Tool will display a warning message informing the user that the operation will be executed on cascade. If the user decides to proceed, the elements having dependencies with that data source will also be deleted. This operation will fail if the user does not have the required privileges to delete these elements.

5.3.3 Creating Base Views from a JDBC Data Source

After creating a data source, it can be used to create base views. For this, you should click on the data source in the list on the left side of the screen to access its contents. In the case of JDBC or ODBC data sources, a tree will be shown with the schemas of the external database. Click on any schema to inspect its tables and their fields (see Figure 7).



Figure 7 Schemas, tables and views of a JDBC source

Follow these steps to create one or more base views:

1. Select the check boxes beside the tables you want to create base views from. 2. Select the project of the new base views using the ‘Target Project’ drop-down.

3. Click on the ‘Create base views’ button beside the ‘Target Project’ drop-down.

After this, the new base views are listed on the left side of the screen hanging from its JDBC data source. In our example, we will create a base view for the phone incidents table ‘phone_inc‘ and another for the internet incidents table ‘internet_inc’. We will select ‘VDPExample’ as the target project. phone_inc provides access to data on the incidents registered in the phone service. Its fields are: the Tax ID (taxId) of the client that reported the incident, an incident identifier (pinc_id), a description (description), the time at which it occurred (ttime), an incident type (inc_type) and an additional specific field (specific_field3). The attributes of internet_inc are the Tax ID (taxId) of the client that reported the incident, an incident identifier (iinc_id), a description (summary), the moment at which it occurred (ttime) and another two additional specific fields (specific_field1 and specific_field2). When selecting two or more tables to create base views, you may need to provide a new name for the views that have the same name as other existing views (see Figure 8)



Figure 8 Renaming new views with the same name as existing ones

When importing only one table at a time (see Figure 9), you have more control over the creation of the base view because you can:

• Change the name of the new base view. • Change the name and type of the new base view’s attributes.

• Select the project of the new base view using the ‘Target Project’ drop-down.



Figure 9 Accepting the schema of a base relation

Note: It is possible to create a base view from a SQL query instead of from a table (‘Create Base View from Query’ button). This option is explained in section 5.5.13.

5.3.4 Viewing the Schema of a Base View

By clicking on a view we have created, we can access the dialog that shows its schema in Virtual DataPort (see Figure 10 and Figure 11 with the schemas of the base views generated from phone_inc and internet_inc). Click on ‘Edit’ to open the ‘Edit view’ dialog. For base views, this dialog allows:

• Renaming the view. • Renaming the view’s attributes. • Changing the types of the view’s attributes.

For derived views, the dialog corresponds to the operation used to create the view (i.e. join, groupby, union, etc). In it, you can:

• Rename the view. • Modify its specific parameters. For example, for join conditions you can change the join conditions,

projected fields, etc. Alternatively, you can replace the view with another one with the same name (see section 5.4.11 for more details about edition/replacement of views). Other possible actions from this screen are:

• Execute: Displays the ‘Execution view’ dialog where the user can execute the default query select * from viewname or add conditions to the WHERE clause. See section 5.4.9 for more details about execution of queries.



• Delete: Delete the view. If there are other derived views using the view in their definition, the tool will

warn the administrator that deleting the view will cause the deletion on cascade of the derived views.

• View VQL: See the VQL code generated to create the view. See section 5.3.4.1.

• Advanced: Opens a dialog to change advanced settings of the view. See section 5.6.

• Used by…: Displays a diagram with the views that depend on the current one. See section 5.3.4.2.

Figure 10 Schema of the base view phone_inc



Figure 11 Schema of the base view internet_inc

5.3.4.1 View VQL

This dialog displays the VQL code generated to create the view. The following options control how the VQL code of the view is generated:

• Show dependencies. If selected, the Tool displays the VQL code to create the view and the elements that the view depends on. Otherwise, it shows only the code to create the view but not the elements that it depends on. For example, if we are viewing the VQL of a view and this option is selected, the Tool will display the VQL code to create the data sources that this view depends on.

• Drop elements. If selected, all the CREATE… statements are preceded by a DROP…CASCADE

statement. Therefore, the generated VQL code will delete, if exists, the element with that name and all the elements that depend on it. E.g. when a server imports a JDBC data source ‘ds_jdbc_acme’, first it will delete the existing JDBC data source ‘ds_jdbc_acme’ and all the views that depend on this data source.

• Replace existing elements. If selected, the statements to create elements will be like this: CREATE

OR REPLACE… As a result, the generated VQL code does not delete the elements that depend on the one that the CREATE statement is creating. E.g. when importing an LDAP data source ‘ds_ldap_activedirectory’ and the server already has an LDAP data source with that name, the Server will replace this data source with the new one. However, it will not modify or delete the views that depend on the existing data source.



• Do not replace existing elements. If selected, the statements to create elements will be like this: CREATE… As a result, if there is already an element with the same name and type, the existing element will not be replaced nor deleted and the statement will fail.

If you want to copy the VQL code, you can click on ‘Copy to clipboard’ to copy all the code of the view. Or, just select the code that you are interested in, right-click it and click on ‘Copy’.

5.3.4.2 Used By…

This dialog displays a diagram with all the views that depend on the current one. On the left side of the dialog, there is a list of all the top-level views that use the current one in their definition. Top-level views are those that are not used to define any other view. Click on one view to display how it is constructed:

• Click on the tree nodes representing combining operations (joins, unions, selections, etc) to display their main properties.

• Click on a data source to view its details.

• Click on a view or a base view to open the dialog showing its schema.

• Click on ‘Save image’ to export this “tree” into an image file.

5.3.5 Importing Web-Service-Type Sources

Continuing with our example, the next step is to create the data source that obtains the sales data. As mentioned earlier, this source is of the Web Service type. To create a new data source representing a Web Service, open the ‘Connect – Projects – VDPExample’ tab and click on ‘New’ beside the ‘Web Services’ tab. The new Web Service data source dialog will be displayed (see Figure 12). The following data are requested:

• Name: Name of the new data source. In our example, the name of the new data source is ‘sales’. • WSDL: Path to the WSDL file containing the Web service specification. We can provide a URL or select a

local file by clicking on ‘Browse’. In our example, the path is $DENODO_HOME/samples/vdp/incidences/sales.wsdl.

• End Point: URL to access the Web service. E.g.

http://localhost:8080/sales/services/RevenueProvider. Usually, the WSDL contains this URL, but sometimes it does not or it does not point to the right location.

o From WSDL: The end point is obtained from the WSDL. o From Variable: The end point is obtained from an interpolation variable. This means that the base

views created over this data source will have an extra field with the name specified in the field ‘Variable’. When querying these views, we have to indicate the end point of the Web service in the WHERE clause of the query. For example, SELECT * FROM v_revenue WHERE endpoint = 'http://acme:8080/sales/services/RevenueProvider' This option is useful when the end point changes regularly or is obtained from another source at

http://localhost:8080/sales/services/RevenueProvider



runtime. See section 5.5.1.3 for more information about interpolation variables.

o Specify: Indicate the URL of the end point in the text field below.

This option is useful when the WSDL does not contain and end point, or it is not correct. E.g. if we deploy the ‘sales’ Web service in a Web container that does not listen to connections at localhost:8080.

• Authentication: Virtual DataPort can access Web services that require authentication. It supports five

authentication protocols: o HTTP Basic and HTTP Digest. [HTTP-AUTH] describes these authentication methods. o HTTP NTLM. Uses the Microsoft's NT LAN Manager (NTLM) Authentication Protocol [MS-

NLMP] to access Microsoft Windows servers. Virtual DataPort supports NTLM v1 and NTLM v2.

o WSS Basic and WSS Digest. Web Services Security [WSS] is a standard for the implementation of security features in applications using Web services. Currently, Denodo supports the authentication profile “Username Token” [WSS-UT].

Instead of providing the credentials when creating the data source, we can assign interpolation variables (see section 5.5.1.3) to the credentials and supply them at runtime. To do that, select the box ‘Obtain credentials from variables’ and fill in the text fields below (‘Name variable’ and ‘Test variable’) E.g., we select ‘HTTP Basic’ authentication, check the box ‘Obtain credentials from variables’ and fill in the field Login ‘Name variable’ with ‘user_name’ and Password Name variable with ‘password’. The views created over this data source will have two extra fields called ‘user_name’ and ‘password’. When querying these views, we have to provide their value in the WHERE clause of the query: SELECT * FROM average_monthly_array WHERE user_name='admin' and password='admin'. The values of the fields ‘Test value’ are used to retrieve the WSDL from the server.

If the HTTP connection with the server is secure (HTTPS), select the ‘Check certificates’ box to accept only the certificates that the Java Virtual Machine considers valid. Otherwise, every certificate will be accepted. See the keytool reference [KEYTOOL] for details on how to import an SSL certificate into the Sun Microsystems JVM.

• Proxy: If the access to the Web Services is established through a proxy, the host name and port where the

proxy is executed must be provided. If it is an authenticated proxy, the user identifier and a valid password must be also specified. It is also possible to use the default http proxy configuration of the server (see section 8.1.5) by selecting the option “Default”.



Figure 12 Creating a Web service data source

5.3.6 Creating Base Views from a Web Service

Once a Web Service data source has been added to the system, it can be used to create base views. For this, you should click on the data source on the list on the left side of the screen to access its contents. In the case of Web-Service-type sources, the tool will show the list of Web Services contained in the WSDL file and the various ports that make up each of them. For each port, the tool will show the list of operations included in it. For each operation, its input parameters, output parameters and input/output parameters are shown (see Figure 13). In our example, the sales Web service has one port with several operations. The operation we are interested in is getAverageMonthlyRevenueBytaxId. It has an input parameter called ‘in0’ that specifies the client Tax ID (taxId) and returns the average monthly volume of sales. There is a ‘Create Base View’ button beside each Web Service operation. If this button is clicked, the system generates a base view from the selected operation. The name of the base view and the name and type of its attributes may be modified at this stage. The project to which the new view will be allocated can be specified using the drop-down ‘Target Project’. In our example, select the ‘VDPExample’ project. In our example, we will rename the base view to ‘average_monthly_sales’, the field ‘in0’ to ‘taxId’ and the field ‘return’ to ‘revenue’.



Figure 13 Operations and parameters of a Web Service

Click ‘Ok’ to create the base view. If the user wishes to create a base view and then import another one from the same data source, he/she can press the ‘OK and import new view’ button, which will create the view and will go back to the screen where the Web Service operation lists is shown. After creating the new base view, it is listed on the left side of the screen below the Web-Service-type data source used to create it. Clicking on the base view, we can access the page that shows its schema in Virtual DataPort (see Figure 14). The available options in this screen were described in section 5.3.4. The results returned by an operation may be of compound data type (e.g. arrays of elements). In that case, these results will be represented in the base view with array and register compound-type fields. It is possible to “flatten” these fields to achieve a more appropriate granularity in the tuples of the base view. This process is described in section 5.4.5. Another similar situation occurs when an operation imported from a Web Service has a compound-type input parameter (e.g. an array of elements). In this case, the input parameters will also be represented in the base relation with array and register compound types. Furthermore, the ROW constructors (to create register-type elements) must be used to build query conditions on this base relation and ‘{ }’ to create type elements. For example: { ROW( ‘parameter1’, ‘parameter2’ ) } specifies one array containing one record with two fields, where ‘parameter1’ is the value of the first field and ‘parameter2’ is the value of the second field.



Figure 14 Schema of the base view average_monthly_sales

5.4 VISUAL CREATION OF VIEWS AND QUERIES

Once the base views have been created, you can access the ‘Combine’ tab to create derived views that combine and integrate data from different base views. After they have been created, derived views can then be used to construct new views or queries. The following sections describe the process of creating views using our example to illustrate the process.

5.4.1 Creating Union-Type Views

A union-type view allows merging the tuples from various input relations in a single view. In standard relational algebra, all the relations must have the same schema (same attributes with the same type). However, in Virtual DataPort an extended union operation is used, whereby if any of the input relations has an attribute that is not present in the others, this is added to the resulting view. To define a union view, open the ‘Toolbox’ menu and click on ‘Union’ . To select the relations that will be part of the union, drag&drop from the list of views and base relations that appear on the left side of the screen or click the name of the view in the ‘Quick Search’ panel on the top of the screen. In our example, we will drag&drop the base views phone_inc and internet_inc (see Figure 15). As input views are added to the union view, the tool automatically generates the schema of the resulting union view. The schemas of the input views are shown on the left of the work area and the output schema is displayed on the right.



Figure 15 Constructing the union view of phone_inc and Internet_inc

Below the input views, the ‘Associations list’ is shown. The fields united by an association are considered a sole attribute in the output schema. In order to establish an association between two fields they must be of the same type. By default, the system will create associations between the attributes of the same name and type (in our example this occurs with the fields time and taxId). Additional associations may also be specified between attributes which receive different names in the different input views. In our example, we will do this with the attributes description of phone_inc and summary of internet_inc. For this, drag&drop from one attribute to another and one sole attribute will appear in the output schema representing both. The default name assigned to the attribute in the output schema will be that of the first attribute marked in the drag&drop operation. Any association can be deleted using the “[x]” button beside the association. The following actions can be carried out on the output view schema (right side of the screen):

• Change the name of the new view • Change the attributes’ name.



• Delete not required attributes: select the checkboxes beside the fields and click the ‘Remove Selected Field’ button.

• Add a WHERE condition to the UNION operation: click on the ‘Set Condition’ button (see section 5.4.3).

• Add derived attributes: click on the ‘New field’ button to open the expressions editor (see section 5.4.3).

We can edit this attribute later by clicking on the icon .

• Remove the duplicate rows from the output: select the ‘DISTINCT clause’ checkbox.

• Order the output by one or more fields: select a field in the ‘ORDER BY’ drop-down and click on the plus icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order.

• Select the project for the new view: select a project in the ‘Target Project’ drop-down.

In our example:

• The new view will be the UNION of the views phone_inc and internet_inc. • We will add another association: internet_inc.summary = phone_inc.description. • The name of the new view will be incidences. • We will eliminate the fields pinc_id, iinc_id, specific_field1, specific_field2

and specific_field3.

• The target project will be ‘VDPExample’. After this, click ‘Ok’ to create the new view. Then, the schema of the new view will be displayed (see section 5.4.6)

5.4.2 Creating Join-Type Views

A join-type view executes the relational algebra operation JOIN on a series of input views. To define a join view, open the ‘Toolbox’ menu and select the ‘Join’ option. To select the relations on which the join operation is to be executed, drag&drop from the list of views on the left side of the screen. In this part of the example, we will combine data obtained from a Web Service with data from a database. To do this, drag&drop the incidences view and the base view average_monthly_sales (see Figure 16). As input views are added, the schema of the resulting join view is generated automatically. The schemas of the input views are shown on the left of the workspace and the output schema is shown on the right. To add a join condition, select an operator in the drop-down list beside the text ’Operator‘ and drag&drop one field of the first view into the target field of the other input view. It is possible to define a more complex join condition, by clicking the ‘Add complex condition’ button. In our example we will select the operator ‘=’ and drag&drop from the attribute taxId of the incidences view to the attribute taxId of the average_monthly_sales view, thus defining the join condition incidences.taxId=average_monthly_sales.taxId.



Figure 16 Constructing the ‘incidences’ and ‘average_monthly_sales’ join view

The join type can also be specified: inner, left outer, right outer or full outer (see ‘FROM Clause’ in the Advanced VQL Guide [VQL]). A conventional ‘inner join’ is used in our example. Note: We can only select ‘Join type’, ‘Method type’ and ‘Order’ if there is at least one ‘Join condition’. Otherwise the new view will be a cross join of the two views. The preferences on the join strategy can also be established. This selection may have a significant impact on the performance of queries (see ‘Query Optimization’ in the Advanced VQL Guide [VQL]). This involves two parameters:

• Execution method (merge, hash, nested or nested parallel). When the nested_parallel method is selected, we also need to fill in the maximum number of parallel subqueries (‘Nested parallel number’) to be run on the second source (see ‘Query Optimization’ in the Advanced VQL Guide [VQL] for further information on the nested parallel method).

• Order in which the input relations must be considered. The view at the top will be considered as a first

relation if the ‘Ordered’ option is selected, and as second, if ‘Reverse Order’ is selected). In our example, no preference will be established for the run strategy, allowing the VDP server to select the one it considers best. The following actions can be carried out on the output view schema (right side of the screen):

• Change the name of the new view



• Change the attributes’ name.


• Add a WHERE condition to the UNION operation: click on the ‘Set Condition’ button (see section 5.4.3)).

• Add derived attributes: click on the ‘New Field’ button to open the expressions editor (see section 5.4.3).





In our example:

• The new view is an INNER JOIN of the views incidences and average_monthly_sales. • The JOIN condition is incidences.taxId=average_monthly_sales.taxId.

• The name of the new view will be ‘incidences_sales‘. • We will delete one of the ‘taxId’ fields of the target view (incidences_sales)

• The ‘Join type’ is ‘Inner’

• The target project will be ‘VDPExample’.

Once the output view has the desired form, press ‘Ok’ to create it. Click on the list on the left-side panel to access the schema of the new view (see Figure 17). Section 5.4.6 contains information about the available options of this dialog.



Figure 17 Schema of the incidences_sales join view

5.4.2.1 Join Conditions with Similarity Operators

Virtual DataPort supports join conditions that use similarity operators. The similarity operator returns a value between 0 and 1 that estimates the similarity between the operands using a certain algorithm. As well as the operands to compare, this operator receives the similarity algorithm to use and a minimum similarity threshold as parameters. Where the similarity between operands reaches the threshold, the condition is true. Otherwise it is false. DataPort includes the operator ~ (Alt-126) to evaluate the similarity between character string-type operands. For this, DataPort includes algorithms based on the edition distance between the texts:

• ScaledLevenshtein • JaroWinkler • Jaro • Level2Jaro • MongeElkan • Level2MongeElkan

It also includes algorithms based on the appearance of words in both texts:

• TFIDF • Jaccard • UnsmoothedJS

Besides, it incorporates combinations of some of these algorithms. I.e: JaroWinklerTFIDF.



If no algorithm is specified, DataPort chooses the one to apply. In that case, only the similarity threshold has to be specified. Example: Consider a variation of the example in which customers are identified by their name instead of their taxId. Unfortunately, the name of the customers in the incidence databases does not exactly match their name in the sales Web Service and, therefore, a join operation with the equality operator will not give good results. In this case, a join can be used with a similarity operator to solve this problem. The incidences example included with the Denodo’s distribution can be used to reproduce this situation. Two additional relational tables are included known as internet_inc_cname and phone_inc_cname. These tables are similar to the internet_inc and phone_inc tables used in the example, although they include a customer_name attribute to indicate the name of the customer instead of the taxId attribute. The sales Web Service also includes an operation known as GetAverageMonthlyRevenueCName that returns a compound data element containing the name and the monthly revenue of all the customers. To solve the problem posed by this example, follow the steps below:

1. Create base views for the internet_inc_cname and phone_inc_cname tables (see section 5.3.3).

2. Create a base view for the GetAverageMonthlyRevenueCName operation (see section 5.3.5). 3. Join the base views created for internet_inc_cname and phone_inc_cname (see section

5.4.1). 4. “Flatten” the compound data element returned by the view created from the

GetAverageMonthlyRevenueCName operation. See section 5.4.5 for a detailed explanation of the “flattening” process.

5. Define a join view using the customer_name field between the view obtained as a result of step 3 and the view obtained as a result of step 4. Specify operator ~ as the join operator and set the similarity threshold to 0.7.

6. Run the view obtained in step 5. The join is made correctly, despite the fact that the customer names in the input views are not exactly the same.

5.4.3 Creating Projection / Selection-Type Views

A Selection-type view executes the relational algebra operations of selection (filtering) and projection (removing fields) on an input view. To define a projection / selection view, open the ‘Toolbox’ menu, select the ‘Projection / Selection’ option and select the input view (drag&drop from the list of views or from the Quick Search panel). In our example, the input view will be incidences_sales (see Figure 18). To add a selection condition, press the ‘Set Condition’ button that will open the conditions editor. This editor has two modes:

• ‘Simple condition’ mode: used to create a list of conditions combined through the AND logical operator, where each condition has an attribute of the view as left operand. By clicking the button beside the ‘Conditions’ label you can add a new condition to the query. For each condition you have to indicate:

o One of the attributes of the view. o The operator of the condition: =, <>, contains, etc o The right operand. It can be any expression using constants, view attributes and functions (see

“Functions for Conditions and Derived Attributes” in the Advanced VQL Guide [VQL]). If the selected attribute is of a compound type or of xml type, click on the button beside the text box to open, respectively, the ‘Compound values editor’ (see section 5.4.3.1) or the ‘XML value editor’.

• ‘Specify WHERE expression’ mode: used to write any condition. This is required, for instance, when you

need to use the OR and NOT operators, or when the left operand of the condition is an expression. Auto-



complete features are available to help constructing syntactically valid expressions. You can also create complex conditions in a purely graphical manner by accessing the Advanced Conditions Editor (see section 12.3).

The condition operators available in DataPort are described in detail in the section “Comparison Operators” in the Advanced VQL Guide [VQL]. The following actions can be carried out on the output view schema (right side of the screen):


• Project an atomic attribute that forms part of a Register element. This can be done by right-clicking on

the required atomic field and selecting the option ‘Project subfield’.


• Add derived attributes: click on the ‘New field’ button to open the expressions editor. In this editor, you can

specify the name of the new field and the expression used to compute it. The expression can use constants, view attributes and functions (see “Functions for Conditions and Derived Attributes” in the Advanced VQL Guide [VQL]). Auto-complete features are available to help constructing syntactically valid expressions. You can also create complex conditions in a purely graphical manner by accessing the Advanced New Field Expression Editor, which is used in a way analogous to the Advanced Conditions Editor (see section 12.3). We can edit this attribute later by clicking on the icon .


• Order the output by one or more fields: select a field in the ‘ORDER BY’ drop-down and click on the plus

icon beside. Once you have added an ORDER BY field, you can choose to order the output in ascending (ASC) or descending (DESC) order.

• If there is a selection condition (‘Set Condition’), select one of the ‘WITH CHECK OPTION’ options to force

every row that is inserted or updated to conform to the definition of this view (see “Use of With Check Option” in [VQL] for more detail). The available options are:

o WITH CHECK OPTION local: the selection condition of this view is checked when a row is

inserted or updated. It does not check the conditions of the lower-level views (the views that participate in the definition of this view).

o WITH CHECK OPTION cascade: the projection conditions of this view and the lower-level views

are checked when a row is inserted or updated.

• Select the project for the new view: select a project in the ‘Target Project’ drop-down list. In our example:

• The input view will be ‘incidences_sales‘ • The name of the new view will be ‘pref_clients_inc_sales‘

• The projection condition will be ‘revenue>600‘ to retain only data of those clients with an average

monthly volume of sales that exceeds 600 euros.



• The target project will be ‘VDPExample’.

Figure 18 Constructing the selection view pref_clients_inc_sales

Once the schema of the output view has the desired settings, press the ‘Ok’ button to create the new view. It will appear in the list on the left part of the screen.

5.4.3.1 Creating Conditions with the Compound Values Editor

When using the ‘Simple condition’ mode to create a condition, we have two options to create constants of compound types (register or array)

1. Write the value directly in VQL syntax. I.e: { ROW( 'B78596011' ), ROW( 'B78596012' ) } See Advanced VQL Guide [VQL] for more details about this syntax.

2. Or, use the Compound values editor. To open it, click on the button beside the text field of the right operand of a condition (only displayed when the selected field is a register or an array)

Next, as an example, we will create a new selection view using the Compound values editor to create a compound-type constant. But first we need to create a new base view:



1. Using the Web Service datasource sales (see section 5.3.5), create a base view based on the getSumRevenueByTaxIds operation. This operation has an input parameter which is an array of strings (the taxId of each client); and returns the sum of the revenue of all specified clients (float-type value).

2. Rename the new base view to RevenueSum.

3. Rename its attributes to clients and totalrevenue.

The clients attribute is an ‘array of registers’ of the type getsumrevenuebyids_in0. Virtual DataPort generates this type automatically when it creates the view. Each element in this array will be a register of the type getsumrevenuebyids_in0_string, which is also automatically generated by DataPort.

4. Rename the element of the clients array to taxId. 5. Click ‘Ok’ to create the base view.

Now we will create a selection view on the RevenueSum base view to obtain the sum of the revenue of the clients with taxIds ‘B78596011’ and ‘B78596012’. The constant operand of the selection condition will be created using the ‘Compound values editor’. The steps to follow are:

1. Select the option ‘Combine – Toolbox – Projection/Selection’. 2. Drag RevenueSum to the work area. 3. Click the ‘Set Condition’ button to access the selection conditions editor.

4. Click the button to add a condition.

5. Select the field CLIENTS and select the operator =.

6. Click the button to open the Compound value editor.

7. Click twice the button beside ‘Value’ to create to elements of the type

getSumRevenuebyTaxIds_in0_string.

8. Click on the new elements to expand them and edit the values of each array element.

9. Enter the value ‘B78596011’ in the field taxId of the first register and ‘B78596012’ in the same field of the second register (see Figure 19)

10. Click ‘Ok’ to confirm the values.

11. Click ‘Ok’ to create the selection view.



Figure 19 Creating a value of type getSumRevenuebyTaxIds_IN0

5.4.4 Creating Aggregation-type Views

An Aggregation-type view allows executing a GroupBy-type operation on an input view. To define an aggregation view, open the ‘Toolbox’ menu, select the ‘Aggregation’ option and drag&drop the input view from the list of views and base views on the left side of the screen (or from the Quick Search Panel) The schema of the input view is displayed on the left of the work area and the current output schema, on the right. Initially the work area will be empty. It is important to remember that an aggregation-type view can have two types of attributes:

• GroupBy Attributes. Attributes that execute the group-by operation. • Attributes resulting from the application of an aggregation expression. Aggregation expressions involve the

use of aggregation functions such as COUNT, MIN, AVG, etc. (see “Aggregation Functions” in [VQL] for the complete list of aggregation functions).

To specify the groupBy attributes drag&drop the fields over the label ‘Group By Fields’ in the lower part of the dialog. An attribute of the input view can also be dragged directly to the output view. In this case, the attribute is considered a group-by attribute and will also appear in the output view schema (see Figure 20).



Figure 20 Creating aggregation view inc_grouped_by_pref_clients

The following actions can be carried out on the output view schema (right side of the screen):


• Delete not required attributes: select the checkboxes beside the fields and click the ‘Remove Selected

Field’ button.

• Add aggregation attributes: click on the ‘New aggregation expression’ to open the aggregation expressions editor. In this editor, you can specify the name of the new attribute and the aggregation expression used to create it. Auto-complete features are available to help create syntactically valid expressions. We can edit this attribute later by clicking on the icon .

• Add a WHERE condition to the projection / selection operation: click on the ‘Set Condition’ button (see

section 5.4.3).

• Add derived attributes: click on the ‘New Field’ button to open the expressions editor (see section 5.4.3). We can edit this attribute later by clicking on the icon .






In our example:

• The input view will be ‘pref_clients_inc_sales‘ • The name of the new view will be ‘inc_grouped_by_pref_clients‘

• The group-by fields will be taxId and revenue.

• Add a new aggregation attribute named num_incidences resulting from the aggregation expression

count(*)

• The target project will be ‘VDPExample’. Once the output view has the desired settings, press the ‘Ok’ button to create it. Clicking on it on the list on the left part of the screen will provide access to its schema.

5.4.5 Flattening of Register-array-type Fields

Denodo Virtual DataPort supports the modeling of data types with tree structure using the types ‘register’ and ‘array’ (see Advanced VQL Guide [VQL]). In Virtual DataPort, an element of the type array can be considered a sub-view; in this way, an array type always has an associated register type that acts like the schema of the sub-view it is modeling. Sometimes you may wish to “flatten” a compound field that contains an array of registers. This is particularly frequent when processing XML- and Web-services-type sources. This section describes how this is done. It is important to note that if you wish to directly “flatten” a register (and not an array of registers), this should be done by creating a new projection/selection view (see section 5.4.3). Going back to the ‘sales’ Web Service data source, we will now use the operation getAverageMonthlyRevenue. This operation does not have any input parameter and returns the sales data from all the clients as an array of objects, where each object has two properties: taxId and revenue (see Figure 21).



Figure 21 The operation getAverageMonthlyRevenue from the Web Service sales

The name of this base view will be average_revenue_array (see Figure 22). It is difficult to combine information from a view that returns an array. So, we will create a new “flattened” view with the two attributes taxId and revenue from the array. The name of the new view will be flat_revenue.



Figure 22 Default base view created for the Web Service operation getAverageMonthlyRevenue

To carry out a flattening operation on a view: 1. Select the option ‘Toolbox - Flatten’ on the menu on the left side 2. Drag&drop the view to be flattened over the workspace. The schema of the input view will appear on the

left part of the workspace and on the right part, the schema of the output view (initially both schemas will be equal).

3. Right-click the array-type fields of the input view and select the ‘Flatten array’ option (see Figure 23).



Figure 23 Flattening the array of the base view average_revenue_array

The output view will then show the flattened schema (see Figure 24).

Figure 24 Output schema of the base view average_revenue_array flattened



In each “flattening” operation it is possible to select just one array. If there is more than one array, this operation must be repeated for each one. The following actions can be carried out on the output view schema (right side of the screen):


• Delete not required attributes: select the checkboxes beside the fields and click the ‘Remove Selected

Field’ button.

• Add a WHERE condition to the projection / selection operation: click on the ‘Set Condition’ button (see section 5.4.3).

• Add derived attributes: click on the ‘New Field’ button to open the expressions editor (see section 5.4.3).





Finally, pressing the ‘Ok’ button creates the new view obtained as a result of the flattening operation (see Figure 25).

Figure 25 Schema of the flat_revenue view



5.4.6 Viewing the Schema of a View

To access the schema of a view, click on the view on the left side of the screen. In addition to the options available for base views (see section 5.3.4), we also have:

• Tree View: see section 5.4.7.

• Data Lineage: see section 5.4.8.

Figure 26 Schema of the view incidences

5.4.7 Tree View Mode

Using the ‘Tree View’ button on the view’s schema dialog displays the view in tree mode. This visualization mode graphically displays the successive levels of views that have been composed to construct this view. For example, Figure 27 shows the tree of the view incidences_sales. The available options in this dialog are the following:

• Click on a view to access the page showing its schema. • Click on the tree nodes representing combining operations (joins, unions, selections, etc.) to display their

main properties. For example, for selection operations, it will display their select condition. • Click on a datasource (leaf nodes) to see its properties (only in administration mode) Sensitive information

such as passwords will not be displayed.



• Click on Save image to export the Tree View into an image file.

Figure 27 Tree view of incidences_sales

5.4.8 Data Lineage

This dialog displays a diagram of the data sources and views used to form the current view and where the information of each field comes from. On the left side of the dialog, there is a list of all the view's fields. By clicking on one of the fields, all the views and data sources that participate in the creation of this field will be highlighted. For instance, if a field f is obtained by evaluating an expression involving two fields f1 and f2 from different data sources, DataPort will highlight the data sources (and their associated views) providing f1 and f2, and the view where the expression to obtain the value of f is defined. The available options in this dialog are the following:

• Click on the tree nodes representing combination operations (joins, unions, selections, etc) to display their main properties.

• View the details of the data sources by clicking on them (only if connected in administrator mode)

• Click on a view or base view to open the dialog showing its schema.

• Click on Save image to export the current Data Lineage Tree into an image file.



Figure 28 Data lineage of incidences_sales

5.4.9 Executing Views

The ‘Execute’ button of the view’s schema dialog, obtains the tuples of the view. The default query is ‘select * from viewname’. By clicking the icon beside the ‘Conditions’ label you can add AND conditions to the query. For each condition you have to indicate:

• One of the attributes of the view. • The operator of the condition: =, <>, contains, etc • The operand. It can be any expression using constants, view attributes and functions (see “Functions for

Conditions and Derived Attributes” in the Advanced VQL Guide [VQL]). If the selected attribute is of a compound type or of xml type, click on the icon beside the text box to open, respectively, the compound values editor (see section 12.3.4) or the xml editor.

It is possible to select the ‘Specify Where expression’ option to write a more complex query. In the text box for writing the query, you can use the auto-complete features to create the expression more easily. When constructing queries, keep in mind that the query conditions should fulfill the view’s mandatory capabilities (see section 5.6.3). The following options are available in this dialog:

• Do not use cache. If selected, this execution of the query ignores the contents of the cache and obtains the data from the original sources even if the cache has been enabled for this view. If this view is a combination of views, the contents of their caches will also be ignored.



• Do not use swap. If selected, this execution of the query will not swap the intermediate results to disk even if swapping has been enabled. If this view is a combination of views, their intermediate results will not be swapped to disk either.

• Limit Rows. Limits the number of rows returned by the query execution.

• Execute with TRACE. If selected, it will enable the ‘View execution trace’ button of the dialog with the

results of the query.

• Query Plan. Provides a look ahead at the execution plan that DataPort will use to run the query (see section 5.4.9.1).

Any query execution can be cancelled at any moment. If results are not being shown yet, cancellation can be achieved by pressing the ‘[X]’ button of the ‘Please Wait’ dialog which can be seen until the first batch of results is shown. Once the query results are being shown, the execution can be stopped by pressing the ‘Stop’ button. Figure 29 shows the result of executing the view incidences_sales. The “View Execution Trace” button allows accessing to the query execution trace (see section 5.4.9.1) once the execution has finished.

Figure 29 Executing the view incidences_sales

If the response to the query execution contains a compound field (registers or arrays of registers), double click on the field to navigate its subcomponents. For example, Figure 30 shows the result of executing the base view average_revenue_array (described in section 5.4.5). This view returns as a result one sole tuple that contains one field of the type array of registers. Each register contains the taxId and the average monthly sales of a client. Double clicking on the field ‘[Array]’ provides access to its contents (see Figure 31). The ‘Back’ button can be used in this screen to return to the higher level.



The data obtained in the execution can be exported to a CSV file by clicking the ‘Save Output’ button. The available options are:

• Select the separator character. • Include the metadata of the query: the VQL command, number of rows, query plan, etc. • Include the columns’ names in the file.

Figure 30 Result comprised of one tuple with a field of the type array of registers



Figure 31 Content of a field of the type array of registers

5.4.9.1 Execution Trace of a Statement

After executing a statement, it is possible to access to a trace of its execution. It is also possible to obtain the execution plan before executing the query. In both cases, the information displayed is the same, except for certain parameters that are only known at run time (e.g. execution time). The statement execution plan is displayed in a tree diagram, where each node represents either an intermediate view involved in the execution of the statement or an access to a data source via a wrapper. If you want to store this diagram into an image file, click on Save image. The most relevant parameters are displayed for each node on the query execution tree. You can copy the value of each parameter by right-clicking on it and then, clicking on Copy or copy the whole trace by clicking on Copy to clipboard. These parameters include:

• Node type. If the node is a view, it indicates the type of the view (base view, union, join, selection/projection, etc.). If it is an access to a data source (wrapper), this indicates the type of source (JDBC, ODBC, Web Service, etc).

• Advanced parameters. These provide further details on each node type. For example: o In the case of wrapper-type nodes, the exact sub-queries executed on each data source and the

connection data used to access each one. o For each view-type node, whether the cache and swapping have been used. o “No delegation cause” explains why a query was not delegated to the source database.

When a query uses two or more views that relay on the same JDBC or ODBC data source, Virtual DataPort will try to delegate the entire query to the source database. This way, it can obtain the whole result executing a single query. When DataPort cannot delegate the entire process of the query to a source database, it will show the icon beside the view that could not be delegated



and the 'No delegation cause' parameter will show the reason why the delegation was not possible. For example, the query may use a function that is not supported by the source database, which will force DataPort to post-process the results obtained. The “No delegation cause” message will help us to rewrite the view so it can be delegated.

• Execution time. Time spent executing a part of the query. • Start time. The exact moment at which node processing began in the execution plan. • End of query time. The exact moment at which node processing (and that of all its children) ended in the

execution plan. • Time obtaining the first tuple of results. Time spent until the node received the first tuple to be processed. • Number of tuples processed. Number of tuples processed by the node. • Status. This indicates whether the node was correctly executed or whether an error occurred.

As an example, Figure 32 shows part of the graphic trace of the execution of the incidences view (created in section 5.4.1). One of the main utilities of the trace function is debugging. In that case, the nodes that caused an error during the execution will be marked red. By clicking on them, more information about the error will be shown.

Figure 32 Incidences view execution trace

Figure 33 shows the execution trace of the incidences_sales join view in the case where the Web Service data source is not accessible.



Figure 33 Join view execution trace with errors

5.4.10 Using VQL Shell

The VQL Shell tool in the VQL Shell tab allows writing and executing VQL statements on the server. It is possible to write one or more statements separated by ‘;’. The statements will be run on pressing the ‘Execute’ button or on pressing the key combination CTRL+ENTER. The VQL Shell provides the following options:

• Stop. Stops the execution of the current query. • Load. Loads a VQL script from a file. The ‘Verbose Output’ option will print the result of the execution of

each VQL statement included in the script. It is also possible to save these results into a file.

• View Execution Trace. Displays the execution trace of the last statement. This option is only enabled if the statement contains the TRACE clause at the end (see section 5.4.9.1).

• Limit Rows. Limits the number of rows returned by the query execution.

• Save output. Export the result of the last executed query to a CSV file. The available options for exporting

are: o Select the separator character. o Include the metadata of the query: the VQL command, number of rows, query plan, etc. o Include the columns’ names in the file.

The VQL Shell includes a “command log” that saves the last VQL statements run. There are two ways of browsing the command log:



• Using the selectable ‘Commands log’. • Using the keys F1 (back to previous command) and F2 (forward to next command).

It is possible to use the menu on the left to speed up writing VQL statements:

• Click on a view to add its name to the current VQL statement. • Right-click a view and the command SELECT * FROM <view name> will appear in the VQL text

box. • Click with the mouse wheel button and DESC VQL <view name> will appear in the text box.

5.4.11 Editing / Replacement of a View Definition

Sometimes, it is necessary to modify a Virtual DataPort view. Note: If it is necessary to edit an existing view because the underlying data source has changed, use the Source Refresh option (see section 5.4.12) In some cases, you can modify a view without regenerating the views that use the modified view. This can be done in two ways:

• Open the schema of the view and click on ‘Edit’. If you are editing a base view, you can change its name and modify the name and type of the view’s attributes. If you are editing a derived view, you can rename it and change the specific configuration of the view. For instance, in join views, you can change the join condition, the join strategy, etc.

• Create a new view with the same name as the view to be replaced. This is useful to:

o Change the operation used to create the view. For example, to convert a join view into a union view.

o Change the data source of a base view. That is, if the data has to be obtained from a different type of source.

When a change in a view affects other views, a dialog will be shown listing the changes before they are applied (see section 5.4.13). If the changes are applied but they are incoherent with the derived views (e.g. on updating a view taking part in a join operation in which any of the join attributes disappears), the icon will be displayed beside the view’s name in the ‘Combine’ menu. If you only need to rename the view, in the left-side menu, right-click on the view and then click on ‘Rename’.



Figure 34 Confirmation of the replacement of a view

5.4.12 Source Refresh

The Source Refresh option updates a base view, reflecting the changes in the underlying data source. To refresh a base view, follow these steps:

1. Locate the base view in the menu Combine > Projects > <project name> > Base Views of the left part of the screen.

2. Open the base view by clicking on it. 3. Click Edit. 4. Click ‘Source Refresh’.

If the modifications in the base view affect other views, the Administration Tool will display a dialog with a list of the affected views. Select which views you want to propagate these changes to, by selecting the check boxes that appears right before each view’s names (select the header's check box to select all the views in the list) After accepting the changes, the views left in an incorrect state appear in the Combine tab marked with a red exclamation mark .

5.4.13 Views Affected by Modifications

Whenever a change in a view affects other views, this dialog will be displayed. It contains a list of all the views that will be affected by these modifications. Click on one of them to see how that view is related to the modified one. If you have added a new field to the view, you can choose if you want to add this new field to all, some or none of the related views.



Figure 35 View affected by modification to another view

5.5 OTHER TYPES OF SOURCES

This section describes the following actions: • Importing data sources of the following types: ODBC, XML, JSON, Web (created using Denodo ITPilot),

Aracne, Google Mini, LDAP, delimited text files and CUSTOM sources. • Creating base relations for these data sources.

• Creating a base relation from a SQL query instead of from a view (only for JDBC/ODBC data sources)

• Configure the data sources’ properties.

When importing data sources of the types XML, JSON or delimited files, it is necessary to specify paths to their files. Section 5.5.1 describes how to do so.

5.5.1 Path Types in Virtual DataPort

When we create a data source of the type XML, JSON or delimited text file (DF), we have to specify a path to the data files. Virtual DataPort supports four types of paths to access a file:

1. Local. Path to a file located in the local file system of the Virtual DataPort server. When you click the ‘Browse’ button, the files listed are the ones located in the system where the server is installed. It does not list the files of the system where the Administration Tool is running.



Instead of specifying the path to the file, we can use interpolation variables (see section 5.5.1.3) if we do not know the path to the file yet. This can be useful if we only know the path to the file at runtime. E.g: we can specify the path ’/tmp/datafiles/@REPORT_ID.txt’. The base views created over this data source will have an extra field called ‘report_id’ and at runtime we will have to provide the value of this field in the WHERE clause of the query.

2. HTTP. Provides access to a file through an HTTP request (GET or POST). The URL can contain interpolation

variables (see section 5.5.1.3)

There are three ways of adding parameters to the body of the POST request (see Figure 36): • URL Parameters: if the URL contains parameters, they will be removed from the URL and sent along

with the body of the request. • XML Body: the XML in the text area will be sent in the body of the request.

• JSON Body: the JSON in the text area will be sent in the body of the request.

Note: You can use interpolation variables (see section 5.5.1.3) into the parameters sent to the server. You will be asked for the value of these parameters when this HTTP connection is tested and when a new base view is created over this data source.

Figure 36 Http POST request with XML Body

The supported authentication methods are: • Basic. The credentials are sent in plain text [HTTP-AUTH]. • Digest. The credentials are sent encoded [HTTP-AUTH].



• NTLM. Uses the Microsoft's NT LAN Manager (NTLM) Authentication Protocol [MS-NLMP] to access Microsoft Windows servers. Virtual DataPort supports NTLM v1 and NTLM v2.

You can use a specific proxy configuration for this data source or use the Default configuration of the DataPort server (see section 8.1.5). If the HTTP connection is secure (HTTPS) select the Check certificates check box to accept only the certificates that the Java Virtual Machine considers valid. Otherwise, every certificate will be accepted. See the keytool reference [KEYTOOL] for details on how to import an SSL certificate into the Sun Microsystems JVM.

3. Denodo Browser. The file will be obtained using the ITPilot's Denodo Browser, which is capable of executing complex navigation sequences written in ITPilot's NSEQL (Navigation SEQuence Language). See the Denodo ITPilot User Guide [ITPILOT] and the NSEQL Manual [NSEQL] for more information about the Denodo Browser and the NSEQL sequences.

4. FTP / SFTP / FTPS. Provides access to a file available in an FTP server. The parameters required to

configure this path are the URL and credentials of the server. If the ‘Test Connection’ box is selected, the server will check if the route is accessible. If you have specified interpolation variables, you will have to provide their values at this point to be able to test the connection. Besides, in delimited file datasources, there are some additional parameters to specify the charset encoding of the file. If the ‘Autodetect encoding’ checkbox is selected, DataPort will try to automatically detect the file encoding through a test connection. If the ‘Autodetect encoding’ checkbox is de-selected, it is possible to choose the desired encoding from the options in the ‘Charset Encoding’ drop-down list.

5.5.1.1 Compressed or Ciphered Data Sources

Virtual DataPort can access data files that have been compressed with ZIP; or have been encrypted with the algorithm ‘Password-Based Encryption with MD5 and DES’ (PBE with MD5 and DES). This encryption method is described in [JCA]. In the new data source dialogs, the available options are:

• Decompress. Select if the data file is compressed. • Decrypt. Select this option if the data file was encrypted with the algorithm PBE with MD5 and DES. In

this case, you have to type the password of the file into the ‘Password’ box.

5.5.1.2 Paths Pointing to a Directory

A Local path to a data source can point to a single file or to a directory. When creating a base view from a data source that points to a directory, DataPort infers the schema of the new view from the first file in the directory. And it assumes that all the other files have the same schema. Note: for XML data sources, if a Validation file has been provided, all files in the directory have to match that Schema or DTD.

5.5.1.3 Paths with Interpolation Variables

Paths can be parameterized depending on the query made by using interpolation variables (see ‘Execution context of a query and interpolation strings’ in the Advanced VQL Guide [VQL]). For instance, the HTTP path:



http://examplesite.com/exampleroute/reports/@{DPT_NAME}.xml uses a variable called DPT_NAME that allows deciding at query runtime the exact file accessed by the path. For instance, if the query specifies as selection condition DPT_NAME = ‘it’, then the accessed URL would be: http://examplesite.com/exampleroute/reports/it.xml If the path specified includes interpolation variables, an intermediate step (see Figure 37) will be required during the base relations creation process, in which Virtual DataPort will request valid values for them. This is necessary so that DataPort can access a data file and obtain the metadata for creating the base relation.

Figure 37 Providing input values to interpolation variables

Each interpolation variable must be associated to a generated base view attribute. If the specified data file includes a field with the same name as the variable, then the variable will be associated with the base view attribute corresponding to this field. Where the SQL query specified returns no field with the same name as the variable, DataPort will add a new attribute to the base relation with the name of the variable. NOTE: When interpolation variables are used in the path description, the “\” character escapes the special characters used to indicate variable names (‘@’, ‘{‘, ‘}’). In Windows systems, this character is also used to specify file paths, causing some ambiguous situations. For example, the system could interpret that the path “c:\tmp\@{var}” references a file named ‘tmp@{var}’ in the ‘c:\’ directory, because the second ‘\’, due to its appearance just before ‘@’, is considered an escape character. However, the “c:\tmp\\@{var}” path would reference a file which name is the value of the var variable and it is found at “c:\tmp”.



5.5.2 ODBC Sources

Virtual DataPort provides access to ODBC sources. Open the ‘Connect – Projects – ProjectName – Relational DB’ tab and click the option ‘New’ beside ‘ODBC’. The new ODBC datasource dialog will be displayed (see Figure 38). The following data are requested in this screen:

• Name. Name of the new data source.

• Database Adapter. Adapter used to connect to the DBMS. If the ‘Choose automatically’ option is selected (recommended), DataPort will discover the most suitable adapter by connecting to the source.

• Connection type. There are two types of ODBC connections:

o DSN (Data Source Name). Use one of the existing DSNs of the system where Virtual DataPort server is installed.

o Direct. Access to a file stored in the local file system of the DataPort server, using one of the ODBC drivers registered on the DataPort server. This is useful, for instance, to access Excel files by using ODBC. The path can use interpolation variables (see section 5.5.1.3).

• Login. User login to access the external database.

• Password. User password to access the external database.

• Choose automatically. If the ‘Choose automatically’ option is marked (recommended), the system will

connect to the ODBC data source to discover the most suitable adapter.

• Test Connection. Virtual DataPort will connect to the source to verify it can be accessed. Click on ’Connections Pool Configuration‘ to configure various parameters of the connections pool that Virtual DataPort uses to optimize access to external databases. The available parameters are the same as in the JDBC pool (see section 5.3.1). Click on ‘DSN Properties Configuration’ to add properties (field/value pairs) to the connection URI.



Figure 38 Importing from an ODBC datasource

Once the ODBC source has been imported, the process for creating base relations is the same as that for JDBC (see section 5.3.3). The Excel ODBC driver provided by Microsoft Windows has some limitations:

- It cannot DELETE data from Excel spreadsheets. It can only query them and append more data. - The Excel driver of MS Excel 2003 or earlier, can execute only one query simultaneously. This can be a

problem if we are combining data from the same Excel file. The newer versions of the Excel driver do not have this limitation.

5.5.3 Multidimensional Database Sources

Virtual DataPort can obtain data from multidimensional databases such as SAP BW3, SAP BI 7 and Mondrian. Note: before creating a multidimensional database source that connects to SAP BW or SAP BI, we have to install the SAP Business Intelligence JDK in the system where Virtual DataPort is running (see appendix 12.9.2) To create a new data source to retrieve data from a multidimensional database, open the ‘Connect – Projects – ProjectName’ tab and click on ‘New’ beside ‘Multidimensional DB’ (see Figure 39). The following data are requested in this dialog:

• Name. Name of the new data source. • Database Adapter. Select the version of the database.

• XMLA URI. The XMLA URI to access the database. E.g:

http://acme.denodo.com:8000/sap/bw/xml/soap/xmla?sap-client=001



• System Name. Name of the instance of the database (only for the SAP BW and SAP BI adapters).

• Language. Language of the connection established with the server (only for the SAP BW and SAP BI

adapters).

• User Name and Password. Credentials to access the database.

Figure 39 Creating a data source to a multidimensional database

Click on ‘Ok’ to create the new data source and then, on ‘Create base view from MDX query’ to create a base view (see Figure 40)



Figure 40 Create a base view from an MDX query (multidimensional data source)

The following data are requested in this dialog:

• The name of the new view. • An MDX query.

Appendix 12.2 explains how the result of an MDX query is mapped to a relational structure.

5.5.4 XML Sources

To create a new data source to access an XML document, open the ‘Connect – Projects – ProjectName’ tab and click on ‘New’ beside ‘XML’. The new XML data source dialog will be displayed (see Figure 41) The following data are requested in this dialog:

• Name. Name of the new XML data source.

• Data Route. Path to the XML file that contains the required data. This path can be parameterized by using interpolation variables (see section 5.5.1.3).

• Validation type. If selected, the structure of the input XML file will be obtained from a Schema or a DTD.

If ‘None’ is selected, Virtual DataPort will analyze the XML document to infer its schema.

• Validate route. Path to the Schema or DTD of the input XML document. This file will be used to obtain the structure of the input XML file.



• Validate data. If selected, Virtual DataPort will validate the input XML file every time the data source is

accessed. The path formats available are described in detail in section 5.5.1.

Figure 41 Importing from an XML data source

Click ‘Ok’ to create the new data source. Then the XML source description dialog will be displayed (see Figure 43). On clicking the ‘Create Base View’ button, an Xpath [XPATH] expression pointing to a node in the XML document can be provided (see Figure 42). This is useful to access only to a portion of the document (this may result in a much faster access to the document). In this case, Virtual DataPort will generate a tuple for each node matching with the path. The subelements of these nodes will be considered as fields in the tuples extracted. If, on the Xpath specification screen, the options RSS or ATOM are marked, the system preconfigures an XPath that is normally suitable for the documents of these types. For example, the preconfigured path for RSS is /rss/channel/item. This path will lead to a tuple being generated in DataPort for each RSS feed item. The subelements of the items will appear as the tuples attributes.



Figure 42 RSS Xpath

NOTE: For performance reasons, the XPath element should be indicated when we are not interested in the full document data. Although it is possible to obtain the same data by creating a base view to access the full XML document and then create projection/selection and flattening-type views to filter the data, this will result in a less efficient access. After configuring the path, the schema of a base relation associated with the source is automatically displayed (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section 5.5.1). The name of the base relation and the name and type of each of its attributes may be changed at this stage. The target project of the new view can be specified with the drop-down ‘Target Project’.



Figure 43 Description of an XML source

By clicking the new base view, we can access the dialog, which displays its schema (see section 5.3.4) Once the base relation has been created, the ‘Edit’ button allows accessing the view edition wizard, where we can modify the name and type of the base relation attributes.

5.5.5 JSON Data Sources

To access a JSON data source, open the ‘Connect – Projects – ProjectName’ tab and click the ‘New’ button beside ‘JSON’. The new JSON datasource dialog will be displayed (see Figure 44) The following data are requested in this dialog:

• Name. Name of the new JSON data source. • Data route. Path to the JSON file that contains the data. This path can be parameterized according to the

query made using interpolation variables (see section 5.5.1.3). The formats of the available paths and how to configure the use of parameters in them are described in detail in section 5.5.1. The most common path type in this case is HTTP.

By clicking on OK, the description screen for the new JSON data source is displayed. By clicking on ‘Create base view’, a new base view will be created and its schema will be displayed (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section 5.5.1.3). The name of the base view and the name and type of its attributes can be changed. The target project of the new base view can be specified with the drop-down ‘Target Project’.



Figure 44 Importing from a JSON datasource

5.5.6 Web Sources (WWW)

Web data sources use wrappers created with the Denodo ITPilot Wrapper Generator Tool [ITPILOT]. Note: We can only use these sources if Denodo ITPilot is installed in the same path as Virtual DataPort. To create a new Web data source, open the ‘Connect – Projects – ProjectName’ tab and click the option ‘New’ beside ‘WWW’ (see Figure 45). This dialog requests a VQL file generated with the ITPilot’s Wrapper Generator Tool (see [GENITP]). We can also deploy Web wrappers into Virtual DataPort straight from the Wrapper Generator Tool.



Figure 45 Importing a WWW wrapper

Click ‘Ok’ to create the new Web source. Clicking on the new data source in the menu of the left side of the screen will open a dialog with the schema of the new data source. The name of the base relation and the type of its attributes can be changed. The target project of the new base view can be specified with the drop-down ‘Target Project’. After doing this changes, click ‘Ok’ to create a base view that will access this new data source.

5.5.7 Delimited File Sources

This type of source access text files in CSV (Comma Separated Values) format or similar. Complex regular expressions can also be used to extract the desired data from other kinds of text files (e.g. application log files).



Figure 46 Importing from a new delimited file source

To create a new data source to access a delimited file (DF), open the ‘Connect – Projects – ProjectName’ tab and click the option ‘New’ beside ‘Delimited Files’ (see Figure 46) The following data are requested in this dialog:

• Name. Name of the new data source. • Data route. Path to the delimited file. The formats of the available paths are described in detail in section

5.5.1. The path can be parameterized according to the query made using interpolation variables (see section 5.5.1.3). NOTE: If the path points to a directory, Virtual DataPort will consider the data from all the files in it (all the files should follow the same data format).

• Column delimiter. Character string to be used as a separator between fields of the same tuple. For

instance, the delimiter can be a semi-colon (;), coma (,), dash (-) or similar. Alternatively, a ‘Tuple Pattern’ can be used (see below).

• Tuple pattern. Regular expression that specifies the format of the tuples that will be extracted from the

delimited file. This regular expression has to match the whole line of the file, not only the part that wants to capture. The syntax of these regular expressions is the one defined by the JAVA language [JREG]. The fields of the views created over this data source will be the capturing groups of the regular expression. Note: the tuple pattern can contain interpolation variables (see section 5.5.1.3).

Example 1 of tuple pattern: Suppose a file that contains product information in the following format (note that the ‘discount’ attribute is optional):



product_name=Acme Laptop Computer;price=1500 euro;discount=50 product_name=Acme Desktop Computer;price=1000 dollar

Figure 47 Example 1 of tuple pattern: contents of the delimited file

The following pattern can be used to extract from each row, the name of the product, its price and currency, and its discount. For the tuples without a discount value, the value of this cell will be NULL.

product_name=(.+);price=([0-9]+)\s(.+);(?:discount=(.+))?

Figure 48 Example 1 of tuple pattern: regular expression to extract the contents of the file

Example 2 of tuple pattern: Suppose we want to extract the name of the files (not directories), its date and its size, from the output of the Windows command DIR: 11/07/2007 10:10 <DIR> .dbvis 09/18/2008 15:09 <DIR> .eclipse 01/19/2011 16:55 <DIR> .gimp-2.6 11/10/2009 18:43 215 .ITLMRegistry 03/26/2010 14:16 3.498 .keystore 05/18/2010 17:56 <DIR> .m2 02/02/2010 15:23 <DIR> .maven 03/26/2010 14:01 <DIR> .netbeans 09/23/2008 15:04 36 .org.eclipse.epp.usagedata.recording.userId02/02/2011 19:20 <DIR> .smc_cache 06/15/2010 09:59 <DIR> .ssh 10/14/2009 13:26 <DIR> .thumbnails 02/15/2010 12:06 0 .Xauthority 01/16/2008 12:02 517 ant.install.log 02/11/2010 13:29 <DIR> Application Data 07/16/2010 08:51 772 build.properties 02/18/2008 15:19 <DIR> Contacts 01/14/2011 10:02 190 default-soapui-workspace.xml 09/16/2010 16:35 78.170.461 denodo-v46-update-201009161734.jar 02/07/2011 11:44 <DIR> Desktop 04/01/2009 15:11 <DIR> Favourites 08/22/2008 12:50 <DIR> Start Menu 01/27/2011 17:18 <DIR> My Documents 02/12/2009 12:36 201 osdadmin.ini 02/12/2009 12:35 201 osdadmin.ini.bak 01/14/2011 10:02 7.958 soapui-settings.xml 02/09/2010 10:02 22.358 temp.txt 04/19/2010 18:55 583 test.cert 02/10/2011 09:22 <DIR> Tracing 03/05/2010 09:41 0 vdpws.log 04/17/2009 09:49 <DIR> workspace

Figure 49 Example 2 of tuple pattern: output of the Windows command DIR

The regular expression we have to put in the ‘Tuple pattern’ field is:

(\d{2}?/\d{2}?/\d{4}?)\s+?\d{2}:\d{2}.*?(\d[^ ]*)\s(.*)

Figure 50 Example 2 of tuple pattern: regular expression to extract the output of the command DIR

The base views created with this tuple pattern will have three fields: the date of the file, its size and its name.



• End of line delimiter. Character string to be used to mark the end of a tuple. The default value is ‘\n’ (carriage return).

• Begin of Data Zone Delimiter. JAVA regular expression identifying the position in the file where the

system must start searching for tuples (or searching for the header if the ‘header’ option was checked). If empty, the search will start at the beginning of the file.

• Include Begin Delimiter as Data. If selected, the text matching the ‘Begin of Data Zone Delimiter’

expression will be included in the search space.

• End of Data Zone Delimiter. JAVA regular expression identifying the position of the file where the system must stop searching for tuples. If empty, the search will continue until the end of the file.

• Include End Delimiter As Data. If selected, the text matching the ‘End of Data Zone Delimiter’

expression will be included in the search space. • Header. If selected, DataPort considers that the first line of the data region contains the names of the

fields in this file. These names will be the fields’ names of the base views created from this data source. • Header Pattern. Regular expression used to extract the name of the fields that form the header. This only

needs to be specified if the header has a different structure than the tuples with data. This option can only be used when the ‘Header’ check box is selected.

Click ‘Ok’ to create the new data source. The delimited files source description dialog will be displayed (see Figure 51). Pressing ‘Create Base View’ creates a base view associated with the new data source (except if the path to the data file includes interpolation variables, in which case there will be an intermediate step. See section 5.5.1.3). The name of the base relation and the name and type of its attributes can be changed at this stage. The target project of the new base view can be specified with the drop-down ‘Target Project’.



Figure 51 Description of a delimited-file-type source

By clicking on a base relation we have created, we can access the page that displays its schema in Virtual DataPort Once the base relation has been created, the ‘Edit’ button allows accessing the view edition wizard, where we can modify the name and type of the base relation attributes.

5.5.7.1 Paths Using Date Ranges

When we need access to files that use a date-based naming convention (as is typical in log files), the DateRange function can be used to consider only the files between a given start date and a given end date. The function admits the following parameters.

- Date format to be used in the queries. - Start date specified in the above format. - End date specified in the above format. - Date format used by the naming convention of the files.

Example: Let us suppose the /tmp path contains the log files daily generated by a certain application. The name of these files has the format conflogyyyyMMdd (e.g. conflog20081119). To add the new delimited files data source, you could specify the path: /tmp/conflog^DateRange("yyyy/MM/dd",@startdate,@enddate,"yyyyMMdd") This way, the base views created from this datasource will include two attributes called startdate and enddate that can be used to specify that a certain query should only consider the files in the specified date range. For instance, the following query:



select * from myview where startdate='2008/11/01' and enddate='2008/11/19'

will obtain data from the log files generated from 01/November/2008 to 19/November/2008. Notice that DateRange can be also used with paths pointing to directories. For instance, let us suppose that the application in the above example stores the log files of each day in a separate directory using the format yyyyMMdd. Then, we could simply specify the path:

/tmp/^DateRange("yyyy/MM/dd",@startdate,@enddate,"yyyyMMdd") In this case, the former query would examine all the files from every directory in the specified date range.

5.5.8 Google Enterprise Search / Mini Sources

Virtual DataPort can use Web search engines built using the Google Enterprise Search / Google Mini software, as data sources [GMINI]. To create a new data source representing a Google Enterprise Search / Google Mini search engine, open the ‘Connect – Projects – ProjectName – Search Engines’ tab and click on ‘New’ beside ’Google Mini’. The new Google Search datasource dialog will be displayed (see Figure 52) The following data are requested in this dialog:

• Name. Name of the new data source.

• Host name. Name of the machine in which the search engine is accessible. This may be a remote machine accessible on the Internet (e.g. ‘search.acme.com’).

• Port. Port on which the Google Mini service is run (usually 80).

• Proxy Configuration. If the connection is established through a proxy, the host name and port where the

Proxy is executed must be provided. If it is an authenticated proxy, the user identifier and a valid password must be also specified. It is possible to use the default http proxy configuration (see section 5.1.5) by clicking the option “Default”.



Figure 52 Importing a new source of type Google Mini

Click ‘Ok’ to create the new data source and see the Google data source description screen (see Figure 53). This dialog lists the fields available in the search engine (these fields are fixed for all Google Enterprise Search / Mini search engines).



Figure 53 Description of the Google Mini source

Clicking ‘Create Base View’ creates a base view associated with the new data source (see Figure 54).



Figure 54 Base View Creation in Google Enterprise Search / Mini Datasources

A value can be specified for the following parameters at the top of the screen:

• Site Collections. Required. It specifies the collections within the Google Enterprise Search / Mini server on which to make the search. The collections are created by the Google server administrator. Its name is upper/lower case-sensitive. It is possible to specify several collections separated by commas. In this case, the search will be made on all of them. The collection that is being searched by a particular search form can normally be obtained by examining the value of the site parameter of its invocations.

• Client. Optional. It identifies the client making the queries. Google Enterprise Search / Mini servers can be

configured to behave differently depending on the client.

• Languages. Optional. If specified, only documents in that language will be returned. The language must be one of the listed in the Google Enterprise Search documentation [GSEARCHLANG].

• Number of Key Match. Optional. Google Enterprise Search / Mini allows the administrator to manually

set a priority of the pages. This parameter receives an integer between 0 and 5, where 5 is the top priority. If this value is established, the searches made on this base relation will only return the pages with the priority specified or over.

The value of these parameters can be modified later, on the advanced configuration dialog of the base view (see section 5.6.7). The name and type of the fields can be modified. The contents of each one of the attributes is:

• TITLE. Title generated by Google server for the document. • SUMMARY. Summary generated by Google server for the document. • URL. Document URL.



• MIMETYPE. MIME type of the document. • RATING. Priority assigned by the Google server administrator for the document. This may take values of

between 0 and 5, where 5 is the maximum priority. • MAXDOCS. Attribute added by DataPort to restrict the maximum number of results returned by a search. • METAS. Attribute of Array of records type that contains the metatags for the document. Each record has

two fields to indicate the name of the metatag (metakey) and its value (metavalue). • CONTENT. Contents of the document. This is the field normally used for searches. • SITE. If specified, Google server will only return the documents that belong to that site / collection (e.g.

‘acme.com’). • FILETYPE. Extension of the document file.

The target project of the new base view can be specified with the drop-down ‘Target Project’. The base view is created just by clicking ‘Ok’. Once the schema has been accepted, the new base relation may be viewed on the left of the screen. The most common way of querying the base relations built from Google Mini sources is using the CONTAINS operator, which runs complex Boolean searches on indexed textual data (see “Support for the CONTAINS operator of each source type” in Advanced VQL Guide [VQL]).

5.5.9 Aracne Sources

Virtual DataPort can create data sources that use search indexes built using Denodo Aracne [ARCN]. To create a new data source representing an Aracne index, open the ‘Connect – Projects – ProjectName – Search Engines’ tab and click on ‘New’ beside ‘Aracne’. The new Aracne datasource dialog will be displayed (see Figure 55). The following data are requested on this dialog:

• Name. Name of the new data source. • Host name. Host where Aracne Index/Search Engine is running.

• Port. Port where Aracne Index/Search Engine is listening to connections. The default port is 9000.

• Login and Password. Credentials to access Aracne Index.



Figure 55 Creating an Aracne data source

Click ‘Ok’ to create the new data source. The Tool will display the schema of the new Aracne data source (see Figure 56). The dialog shows the indexes contained in the Aracne server and the fields included in each one.



Figure 56 Description of the Aracne source

Click on ‘Create Base View’ to create a new base view associated with the index selected (see Figure 57). The name of the base view and the name and type of its attributes can be changed at this stage. Non-required attributes can be deleted. The base views pregenerated from Aracne indexes will have an attribute of the appropriate type for each field in the Aracne Index. Usually, Aracne indexes are used to index the data extracted by Denodo Scheduler jobs and their schema depends on the type of executed job (see the Denodo Scheduler Administration Guide [SCHED] for further details). Figure 56 shows the fields in an index created by the documents exported by a Scheduler job of type ARN.



Figure 57 Creating base views in Aracne sources

Alongside each attribute of the base view, there is the “Create Main Terms” option (see subsection 5.5.9.1). The target project of the new base view can be specified with the drop-down ‘Target Project’. The base view is created by clicking ‘Ok’. If the user wants to create a base view and then import another one from the same data source, he/she can click ‘OK and import new view’, which will create the view and will go back to the screen where the Aracne server’s indexes are listed. Once the schema has been accepted, the new base view may be viewed on the left of the screen. The most common way of querying the base relations built from Aracne sources is using the CONTAINS operator, which runs complex Boolean searches on indexed textual data (see “Support for the CONTAINS operator of each source type” in Advanced VQL Guide [VQL]).

5.5.9.1 Accessing the Most Relevant Terms of a Document

Denodo Aracne is capable of automatically generating the most relevant words of a document or one of its fields, according to the TFIDF (Term Frequency Inverse Document Frequency) relevance measurement. These terms can be accessed via Virtual DataPort, as described in this section. The most relevant terms are accessed as new attributes in the DataPort base view. To create a new attribute in the base view containing the most relevant terms of the value of a certain document field, click on the ‘Create main terms’ option alongside the required field. For example, if you want to add a new attribute containing the most relevant terms of the SEARCHABLECONTENT index field, click on ‘Create main terms’ alongside it. Figure 58 shows this new attribute (known as SEARCHABLECONTENT_MAIN_TERM).



Figure 58 Adding an attribute with the most relevant terms of the SEARCHABLECONTENT field

The new attribute SEARCHABLECONTENT_MAIN_TERM will contain an array of records. Each record in the array contains two fields:

• The relevant term. This field takes by default the name of the index field, adding the suffix _TERM (in this case, therefore, the name will be SEARCHABLECONTENT_TERM). However, as can be seen in the figure, the name of the field can be changed.

• Its position in the list of the most relevant. This field takes by default the name of the index field, adding the suffix _SCORE (in this case, therefore, the name will be SEARCHABLECONTENT_SCORE). However, as can be seen in the figure, the name of the field can be changed. The most relevant term takes position 1.

In the new attribute creation process, it is possible to specify two parameters:

• Number of main terms. Maximum number of relevant terms to be included for each document. • Filter main terms words. List of “usual words” (separated by commas) that must not appear among the

most relevant terms for this field. If the list of relevant terms generated by Aracne includes any of those, they will be removed from the list. It is important to note that only usual words specific to the application must be specified. The usual words in the language used such as articles, pronouns, etc. (commonly known as “stopwords”) are already eliminated by Denodo Aracne. The list of usual words may be contained in a file. The file specified must be a text file, where the words will be separated by commas.

Furthermore, at the bottom of the base view creation screen it is displayed the ‘General Words to Filter Main Terms’ parameter that indicates a list of usual words common to all fields in the base view. Once again, you do not have to worry about specifying usual words in the language used such as articles, pronouns, etc. (commonly known as “stopwords”), as they are already eliminated by Denodo Aracne.



The value of these parameters can be subsequently modified on the advanced configuration screen of the base view created (see section 5.6.7).

5.5.10 LDAP Sources

The LDAP-type data sources allow registering an LDAP server in Virtual DataPort. LDAP-type data sources can be used for two purposes:

• To extract data stored in the LDAP server. • To delegate authentication tasks to an LDAP server:

o Authentication of users of Virtual DataPort servers (see section 9.3.3 for further details on the creation of users in DataPort)

o Authentication of users of Web Services published by the Virtual DataPort server (see section 6) To create a new data source representing an LDAP server, open the ‘Connect – Projects – ProjectName’ tab and click the option ‘New’ beside ‘LDAP’. The new LDAP datasource dialog will be displayed (see Figure 59). The following data are requested on this screen:

• Name. Name of the new data source. • Server URI. Path to the LDAP server. For example: ldap://acmehost:389 • Login and Password. Credentials to access the LDAP server.

• Select Use Paging if the LDAP server has a limit on the number of results per query. If selected, Virtual

DataPort will do paged searches to obtain all the results of the queries.

• Select Test connection to test the connection with the LDAP server before creating the data source.



Figure 59 Creating an LDAP data source

After clicking ‘Ok’, the Tool will display a dialog containing a list of the different object classes that the LDAP server contains (see Figure 60)



Figure 60 New LDAP data source

Select the desired object classes and click on the ‘Create base view’ button. Then, Virtual DataPort will create a new base view with the fields of the selected classes and display it to the user. If required, the name of the base view and the name and type of its attributes can be changed at this stage. The target project of the new view can be selected with the box ’Target Project’. Click on ‘Ok’ to create the base view. Optionally, before clicking on the ’Create base view‘ button, it can be indicated whether the query should: recursively consider all the nodes in the subtree hanging from the node pointed by the datasource path, or it should only consider the direct children of the node. Once the new data source has been created, it is also available to create new users (see section 9.3.3). It is also possible to create a base view from an expression which is delegated to the LDAP server. For this, you must click on the ’Create base view from expression‘ button, and the administration tool will show a form to introduce the expression (see Figure 61). This expression can have interpolation variables (see section 5.5.1.3). If interpolation variables are used, then the user will be asked for example values for the variables to perform a query against the LDAP server. Using this query, the system will obtain the object classes that can be reached using the expression. The user can then select the object classes to be used to generate the base view schema (see Figure 62). Optionally, it can be specified whether the query to perform against the LDAP server should be recursive or not. The interpretation of this option is the same as in the formerly explained case of creating the base view without specifying an expression.



Figure 61 New LDAP base view from expression



Figure 62 Accessible objects of an LDAP base view created from an expression

5.5.11 BAPI Sources

Virtual DataPort can invoke SAP BAPIs (Business Application Programming Interfaces) to obtain data stored in SAP ERP and other SAP applications. Important: before creating any BAPI data source, we have to install the SAP Java Connector 3 in the system where the Virtual DataPort server is running (see Appendix 12.9.1) To create a new BAPI data source, open the ‘Connect – Projects – ProjectName’ tab and click on ‘New’ beside ’BAPI’. The new BAPI data source dialog will be displayed (see Figure 63)



Figure 63 Creating a new BAPI data source

The following data are requested in this dialog: • Name. Name of the new data source. • System Name. The SAP system ID of SAP ERP.

• Host. Host where SAP ERP is running

• Client ID. Client id.

• System Number. The two-digit number to differentiate two SAP ERP running on the same host.

• User Name and Password. Credentials to access SAP ERP.

Click on ‘Ok’ to create the new data source. Then, click on ‘Create base view from BAPI’ to create a base view (see Figure 64)



Figure 64 Creating a new BAPI (SAP ERP) data source

Fill in the name of the BAPI and click on ‘Ok’ to display the schema of the result of the BAPI invocation. We can change the name of the new view and its fields and after that, click ‘Ok’ to create the view.

5.5.12 CUSTOM Sources

CUSTOM-type data sources (also called MY data sources) are ad-hoc wrappers that have been specifically developed to access a specific source and can be used in Virtual DataPort. See the VQL Advanced Guide to learn to develop CUSTOM wrappers. In the path $DENODO_HOME/samples/vdp/customWrappers there are examples of CUSTOM wrappers. The README file in this path contains instructions on how to compile, install and use them. The screenshots displayed in this section use the sample CUSTOM wrapper com.denodo.vdp.demo.wrapper.custom.MetaPayRollWrapper.



Figure 65 Importing a CUSTOM wrapper

To create a new data source of this type open the ‘Connect – Projects – ProjectName’ tab and click the option ‘New’ beside ‘Custom’. The new Custom datasource dialog will be displayed (see Figure 65). The following data are requested in this screen:

• Name. Name of the new data source. • Class name. Full name of the JAVA class that implements the CUSTOM-type wrapper.

• Classpath. If the .jar library was not imported to the server using the ‘Jar Management’ option (see

section 8.3), you have to indicate its location by clicking the ‘Browser’ button and selecting it. Otherwise it can be left empty.

• Select Jars. If the class belongs to an imported Jar file, select this checkbox and select the appropriate

Jar. Note: to select two or more Jars, hold the key Ctrl and click on the required Jars. This may be necessary if the custom wrapper relies on external jars that have also been imported into Virtual DataPort.

Where the examples of CUSTOM wrappers included with DataPort have been compiled and installed, follow the instructions and click on ok to create the new data source (see Figure 66).



Figure 66 New CUSTOM data source

Click on the “Create Base View” button to create a base view associated with the new data source. The CUSTOM wrapper used to implement access to the source can accept input parameters (see “Creation of Custom Wrappers” in the Advanced VQL Guide [VQL]). In Figure 67, the user has to fill in the values of the following parameters:

• ENTERPRISE = enterprise1 • YEAR = 2007



Figure 67 Creating a base view from a CUSTOM data source

By clicking on ‘Ok’, the schema of a base relation associated with the data source is created (see Figure 68).



Figure 68 New base view from a CUSTOM data source

Select the project assigned to new base view in the drop-down ‘Target Project’. By clicking on the base view, we can access the page which displays its schema. Once the base relation has been created, click on the ‘Edit’ button to access to the view edition screen, which allows renaming attributes or changing their types.

5.5.13 JDBC/ODBC sources through SQL Query

Sometimes, it may be necessary to create a JDBC/ODBC base view using an arbitrary SQL query, instead of using a table of the database as a “template”. This can be useful in two scenarios:

• When we need to execute a database stored procedure or a query that uses a stored procedure, to obtain the results.

• When it is faster to create a base view specifying an SQL query, than importing all the base tables and

creating the equivalent combination view. To create a SQL Query-type base relation, use the ‘Create Base View from Query’ button on the JDBC or ODBC data source detail screen. You will access the screen shown in Figure 69.



Figure 69 Creating a base view using an SQL query

Once the name for the base relation and the required SQL query have been specified, pressing ‘ok’ creates the schema of the new base relation. Its schema can be edited as usual. The SQL query used to create the base view may also use interpolation variables (see section 5.5.1.3), which allow the SQL query made on the source to be parameterized according to the specified query conditions. In this case, after pressing ‘Ok’, the user has to fill in the wizard with the values of the variables used (see Figure 70). This is necessary so that DataPort can execute the query to obtain the metadata required to create the base relation. Each interpolation variable must be related with an attribute belonging to the generated base relation, so that the variable can obtain its value at run time. If the SQL query used to access the data source returns any field with the same name as the variable, then the variable will be associated to the corresponding base relation attribute. In case the SQL query does not return any field with the same name as the variable, DataPort will add a new attribute to the base relation, with the same name. If the SQL query invokes a PL/SQL function using an interpolation variable, you need to assign an alias to the return value. For example:

SELECT function_test(@VAR1) AS value FROM internet_inc



Figure 70 Editing the value of the interpolation variables

5.5.13.1 Using the WHEREEXPRESSION Variable

As we commented in the previous section, the SQL Query used to create the base relation can contain “interpolation variables” (see section 5.5.1.3 and the VQL Advanced Guide [VQL]]), to parameterize the SQL Query executed. There is a predefined interpolation variable called WHEREEXPRESSION that simplifies the process of creating base relations by using the SQL Query method. Besides, the use of WHEREEXPRESSION can also help optimize the execution. Specifically, if a join view is using the NESTED execution method (see the “Query Optimization” section in the VQL Advanced Guide [VQL]), and the second view in the join operation is a ‘SQL Query view’, it is highly advisable to create this second view using the WHEREEXPRESSION variable. This allows DataPort to apply optimization techniques that would not be possible otherwise. The WHEREEXPRESSION variable can be used in the SQL query as substitute of all or part of the WHERE clause in the query. At run time, DataPort will substitute the variable by a valid query condition built from the query conditions received by the base relation. For example, let’s suppose that we create a base relation called VIEW1 from the following SQL query: SELECT StorProc(FIELD1), FIELD2, FIELD3, FIELD4 AS ALIAS4 FROM TABLE1 WHERE @WHEREEXPRESSION

Notice that the query uses a stored procedure in the SELECT clause, so it is necessary to create the base view by using a SQL query.



After clicking ‘Ok’, the user will have to provide the values for the variables used, so DataPort can execute a query to obtain the metadata required for creating the base view (see Figure 70). In this example, any valid query condition could be used as value for WHEREEXPRESSION like, for instance: FIELD2=’f2’ AND FIELD4=’f4’

Once DataPort has obtained the required metadata by executing the sample query, it will show the list of base view attributes that will be generated (see Figure 71). The user may specify information about any aliases used in the query SELECT clause. For instance, in the previously commented query, the ALIAS4 alias is defined, which is associated to the FIELD4 field. Therefore, the user must specify the FIELD4 value for the attribute named ALIAS4. After providing the alias information, the base view will be created.

Figure 71 Specifying aliases

In our example, let’s consider what would happen if, once the VIEW1 view has been created, the following VQL query is executed (NOTE: in the example, it is assumed that the user has not modified the attribute names when creating the base relation, and, therefore, they match the ones specified in the SQL query used to create the base relation): SELECT * FROM VIEW1 WHERE FIELD2=’f2’ AND ALIAS4=’f4’

In this case, DataPort will substitute the WHEREEXPRESSION variable at run time, by the value required to execute the equivalent query on the original database. In this case:

SELECT StorProc(FIELD1)AS ALIAS1, FIELD2, FIELD3, FIELD4 AS ALIAS4 FROM TABLE1 WHERE FIELD2=’f2’ AND FIELD4=’f4’



5.5.14 Data Source Configuration Properties

Data Source Configuration properties allow specifying certain characteristics of the underlying data sources such as the operations they support. Knowing the capacities of each data source is important for optimization reasons since it allows Virtual DataPort to delegate to the data source as much processing as possible to optimize response times and minimize traffic through the network. NOTE: Typically, users do not need to edit this information since DataPort automatically uses suitable configurations for most common data sources. The properties of each data source can be configured by selecting the required data source and pressing the ’Edit’ button before clicking on the “Source Configuration” link. This link only appears in data sources with configurable properties. The configurable properties are as follows: NOTE: Virtual DataPort has default values for some specific relational databases (MySQL, Oracle, Postgres, etc) that vary in relation to the default values mentioned below.

- Delegate All Operators (DS: JDBC, ODBC). This indicates whether the source allows for all operators to be delegated. The value is “false” by default.

- Delegate Array Literal (DS: JDBC, ODBC). This indicates whether the source allows for array-type compound constants to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Compound Field Projection (DS: JDBC, ODBC). This indicates whether the source allows projections on compound fields to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate GROUP BY (DS: JDBC, ODBC). This indicates whether the source allows the GROUP BY clause to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate HAVING clause (DS: JDBC, ODBC). This indicates whether the source allows the HAVING clause to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Inner Join (DS: JDBC, ODBC). This indicates whether the source allows for the Inner Join operator to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Join (DS: JDBC, ODBC). This indicates whether the source allows for the Join operator to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Left Function (DS: JDBC, ODBC). This indicates whether the source allows for conditions with functions on the left part to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Left Literal (DS: JDBC, ODBC). This indicates whether the source allows for conditions with constants on the left part to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Natural Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Natural Outer Join operator to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Delegate NOT Condition (DS: JDBC, ODBC). This indicates whether the source allows the NOT condition to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate OR Condition (DS: JDBC, ODBC). This indicates whether the source allows for the OR condition to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate ORDER BY (DS: JDBC, ODBC). This indicates whether the source allows the ORDER BY clause to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Projection (DS: JDBC, ODBC). This indicates whether the source allows for projections to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Register Literal (DS: JDBC, ODBC). This indicates whether the source allows for register-type compound constants to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Delegate Right Function (DS: JDBC, ODBC). This indicates whether the source allows for conditions with functions on the right part to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Right Literal (DS: JDBC, ODBC). This indicates whether the source allows for conditions with constants on the right part to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate Selection (DS: JDBC, ODBC). This indicates whether the source allows for conditions to be delegated. The value is “true” by default for JDBC and ODBC sources.

- Delegate UNION (DS: JDBC, ODBC). This indicates whether the source allows for the union operator to be delegated. The value is “true” by default for JDBC and ODBC sources.



- Supports Modifier in Aggregate Function (DS: JDBC, ODBC). This indicates whether the source supports DISTINCT/ALL modifiers in aggregate functions.

- Supports Branch Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for (left | right) outer join. The value is “false” by default for JDBC and ODBC sources.

- Supports Eq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Equality Outer Join operator to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Explicit Cross Join (DS: JDBC, ODBC). This indicates whether the source allows for the Explicit Cross Join operator to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Full Eq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Full Equality Outer Join operator to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Full NotEq Outer Join (DS: JDBC, ODBC). This indicates whether the source allows for the Full Not Equality Outer Join operator to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Fusing in Using AND Natural Join (DS: JDBC, ODBC). This indicates if the source merges the same fields when running a natural join or a join with the USING clause. The value is “false” by default for JDBC and ODBC sources.

- Supports Join On Condition (DS: JDBC, ODBC). This indicates whether the source allows for the Join On clause to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Natural Join (DS: JDBC, ODBC). This indicates whether the source allows for the Natural Join clause to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Supports Using Join (DS: JDBC, ODBC). This indicates whether the source allows for the Using Join clause to be delegated. The value is “false” by default for JDBC and ODBC sources.

- Delegate Aggregate Functions List (DS: JDBC, ODBC). This indicates the aggregation functions that can be delegated. In JDBC and ODBC sources, the list is made up of the AVG, COUNT, MAX, MIN and SUM functions.

- Delegate Scalar Functions List (DS: JDBC, ODBC). This indicates the scalar functions that can be delegated. - Delegate Operators List (DS: JDBC, ODBC). This indicates the operators that can be delegated. In JDBC and

ODBC sources, the list is made up of the =, <>, <=, >, >=, in, between, contains, containsor, like, isnull, isnotnull, istrue and isfalse operators.

5.6 ADVANCED CONFIGURATION OF VIEWS

The ‘Advanced’ button on the schema screen of a view provides access to its advanced configuration wizard (see Figure 72). From this screen, you can configure the view cache, its query capabilities, its swap to disk policy, certain characteristics of its execution plan (only in derived views), its internationalization configuration (only in base views) and some specific aspects for views from some types of sources. The following subsections deal with each of these aspects.



Figure 72 Advanced Configuration screen of the view average_monthly_sales

5.6.1 Internationalization Configuration

The internationalization configuration of a base view specifies aspects such as time zones, languages or currencies. For example, the currency to be used in the values of the type Money is specified by this property. DataPort includes the most usual internationalization configurations, but also allows new configurations to be created ad hoc. See “Creating New Internationalization Configurations” in the Advanced VQL Guide [VQL] for more information.

5.6.2 Configuring the Cache of a View

The following parameters may be used to configure the view cache (caching must have been previously enabled in the server (see section 8.1.3):

• Time to live. Indicates the time to live (in seconds) for the data stored in the view cache. If the ‘Default’ checkbox is checked, the default value defined in the server configuration panel will be used (see section 8.1.3).

• Cache on/off/post. The cache can be activated or deactivated for the view using the cache on/off/post selector. If the option ‘post’ is selected, the system will detect whether a given query can be answered using another previous query, even where this is not exactly the same as the new query. For example, if the results of a previous query select * from view where field1 = a are in the cache and the system receives the query select * from view where field1 = a and field2 = b, it can be responded to on the basis of the results of the first query, applying a post-processing operation that returns only those tuples where field2 = b. If the on option is selected, the system only uses the cache if the query received is the same as the preceding query.



Use of the post-processing technique may not be desirable if a wrapper does not always return all results for a query made to a specific source. For example, if a wrapper that accesses a Web source returns only the first 100 results returned by the source for the query select * from view where field1 = a, then the result of applying the post-processing condition (field2 = b) to the results of the query may be different to the result obtained when executing directly on the source select * from view where field1 = a and field2 = b.

It is also possible to invalidate the cache content for a view. There are the following options:

• Invalidate all. Remove all the cache content for the view. • Invalidate partially. Remove only the content associated to queries that verify a set of specified conditions.

These conditions are created with the conditions editor. • Invalidate on cascade. Also, remove the cache of the lower-level views used to create the edited view. This

option is supported for both total and partial invalidations. See section 4.5 for information about deleting the cache content of more than one view at the same time.

Figure 73 Cache invalidation of a view

5.6.3 Swapping Configuration

To avoid memory overflows, Virtual DataPort can swap the intermediate results generated during the execution of a query. It can also swap the intermediate results of the sorting operations (views with ORDER BY).



Before enabling swapping in a specific view, we have to enable the swapping engine in the server (see section 8.1.6 for information on how to do it) When a query is being executed, Virtual DataPort will swap to disk when an intermediate result produced during the execution exceeds the maximum size specified. This size can be specified (in megabytes) using the “Max size” field. Generally, the specified value should not be greater than one third of the memory available for the JAVA virtual machine on which the DataPort server is run. If the ‘Default’ checkbox is checked, the default value defined in the server configuration panel will be used (see section 8.1.6). NOTE: DataPort can also configure the use of “swapping” dynamically for a specific query using the VQL CONTEXT clause (see “Context clause” in the Advanced VQL Guide [VQL]).

5.6.4 Editing the Execution Plan

A key aspect of query and view run optimization in DataPort is the execution strategy used for each join operation involved. As described in section 5.4.2, it is possible to specify the required execution strategy on creating a join-type view. An execution strategy for a join operation indicates two parameters (see ‘Query Optimization’ in the Advanced VQL Guide [VQL] for more details):

• Run method (merge, hash, nested or nested parallel) and • Order in which the input relations must be considered. The view at the top will be considered as a first

relation, if the Ordered option is chosen, and as second, if Reverse Order is chosen). In addition, it is also possible to choose the ANY option, so DataPort will choose the strategy to apply. However, a join view can be used in defining new derived views and the optimal execution strategy may be different in each case. Example: Suppose there is a join view V3 with attributes A, B and C. V3 has views V1 (with attributes A and B) and V2 (with attributes B and C) as input relations. Let us also suppose that queries on V3 are using the nested execution method. Common optimization practice on using this execution method involves ordering the input relations so that the first is the one to provide least tuples to the join operation. In our example, suppose that the number of input tuples to the join from V1 is normally less than that of V2 and, therefore, V3 is usually queried by considering V1 as the first relation. Now suppose that you want to create a new view V4 that is a selection view on V3 by applying the condition C=c. Also suppose that there are few tuples in V2 that verify this condition. Then, on running a query on V4, the most common situation will be for the join view V3 to be given less tuples from V2 than from V1. Therefore, when V4 is run, V2 should be considered the first relation, unlike the general case where the opposite is sought. To deal with these cases, in any derived view V it is possible to define the preferred run strategy for each join view involved in its definition. These strategies will only be used to run queries on V and will not overwrite the default execution preferences of these join views.



Figure 74 Editing the Execution Plan of a view

Hence, the preferences for the execution plan of V will not affect the queries made directly on the original join views or on other derived views using them. To edit the preferences on the join execution strategy for a derived view, access the advanced configuration screen and click on the ‘Edit Execution Plan’ button. The execution plan edit screen will then be displayed for the view (the execution plan edit screen is shown in Figure 74 for view inc_grouped_by_pref_clients created in section 5.4.4). The screen displays the view definition tree. For each join operation involved in the definition, it is possible to mark the ‘Specify execution plan (do not inherit it)’ box to enable two selectable fields to choose the execution method and the order of the input relations required. If the ‘ANY’ option is chosen for a parameter, DataPort will use internal cost information to choose the most suitable option. On clicking ‘ok’, you will return to the advanced configuration screen for the view. By clicking ‘ok’ again, the configuration changes made on this screen will become effective. NOTE: DataPort can also configure the execution plan preferences for a specific query using the VQL CONTEXT clause (see “Context clause” in the Advanced VQL Guide [VQL]).

5.6.5 Query Capabilities

Some data sources such as Web sites or systems that offer a Web Service interface do not allow any query to their data, instead they present interfaces with limited query capability (e.g. HTML forms in the case of a Web site and a list of parameters in the case of a Web Service operation). For this reason, Virtual DataPort allows the administrator to explicitly specify the query capabilities of the base relations. Also, Virtual DataPort automatically calculates the query capabilities of the derived views according to the capabilities of its base views.



Therefore, when the advanced configuration of a view is accessed, the available options vary according to whether the view is a base relation (also called base views) or a derived view. The difference lies in that in the case of base relations the query capabilities can be modified, while in the case of derived views they can be seen but not directly modified. The query capabilities are expressed using a list of search methods. Each search method specifies a different way to query the view (or base relation). See the next section for a detailed description of how to interpret search methods of a view (or base relation). When a base relation is graphically created, the system automatically assigns search methods to it. While in the majority of cases the methods automatically created do not need to be modified, this may not always be the case, particularly in sources of the type Web and Web Service. Once in the advanced configuration view of a base relation, the default search methods generated for same may be changed easily (see Figure 72). As can be seen in the figure, a search method specifies for each attribute of the view (see the subsection below for more details):

• Its data type, • A list of operators for which the field can be queried. ‘any’ indicates any operator. New operators may be

added for the attribute using the drop-down menu in its ‘Operators’ column. Operators may also be eliminated using the buttons ‘[x]’ that appear beside each operator.

• Whether it is obligatory (‘OBL’), optional (‘OPT’) or not supported (‘NOS’) in the queries to the view. • The multiplicity supported for the attribute in queries to the view. • A list of the possible values for the attribute that may appear in a condition of a valid query to the view. If

no value is specified, it is assumed that every value of the attribute data type is valid. To add new values simply enter them into the text box that appears in the ‘Values’ column of the attribute and press the button that appears on the right. A value may be deleted using the button ‘[X]’ that appears at its side.

• Whether the attribute appears in the response of the queries made to the view. The VQL language may be used directly to add or delete complete search methods (see the Advanced VQL Guide [VQL]).

5.6.5.1 Search Methods

Each base relation explicitly describes its query capabilities through what are known as search methods. If a relation has no search method, then no query can be made to it. Each search method is comprised of a series of 5-tuples. Each 5-tuple represents a restriction that a specific query should comply with to be executed on the source using this search method. In this sense, a search method that has no 5-tuple allows every query (this is the case of sources that have no limitations in their query capabilities such as, for example, conventional relational databases). The format of a 5-tuple is (attribute, operators, obligatoriness, multiplicity, possible_values) where:

• attribute is an attribute of the relation.

• operators is the group of operators that can be used in conditions with this attribute in the queries to this view and with this search method. ‘ANY’ represents any operator allowed by the attribute data type.



• obligatoriness can have three values: ‘OBL’ indicates that the attribute should mandatorily appear in any query to the source. ‘OPC’ indicates that the attribute may or may not appear in the query (it is optional) and ‘NOS’ indicates that the queries for this attribute are not allowed in the source.

• multiplicity indicates how many values can be included in the source query for the given attribute and operator. If it is not possible to make queries for this attribute (“NOS” value in the obligatoriness field), the value is necessarily 0. ‘ANY’ indicates that a query, for any given attribute and operator, could have any number of values.

• possible_values is the list of values that can be used to query the attribute. If the value ‘ANY’ is contained in it, this means that the search range is not limited (within the range associated with the attribute data type), and the attribute can be queried about any value. If the obligatoriness field is set in the 5-tuple to the ‘NOS’ value, then it necessarily takes the value of an empty set.

Example: Consider the example of a virtual bookshop on the Internet offering the search form of shown in Figure 75.

Figure 75 Search form for a bookshop

The form obliges the user to specify a value for the TITLE attribute and gives the option to set a value for the AUTHOR attribute and for the FORMAT attribute (restricted to a group of values). Searches by title and author are searches by keyword (operator like). A search by exact phrase (operator =) is indicated by selecting the box next to the search box of the field. For each attribute, a search can specify one value only. In addition to the fields TITLE, AUTHOR and FORMAT, the shop returns as output a PRICE attribute, which cannot be queried using the form. Let us model this source as a relation R={TITLE,AUTHOR,FORMAT,PRICE} with a search method containing the 5-tuples shown in Figure 76.

(TITLE,{like,=}, OBL, 1, Any) (AUTHOR, {like,=}, OPT, 1, Any) (FORMAT, {=}, OPT, 1, {‘All formats’,‘Hardcover’,‘eBooks’, Paperbacks’}) (PRICE, {}, NOS, 0, {})

Figure 76 Search method for a bookshop

Note that the first 5-tuple has the value {like, =} in the OPERATORS field and OBL in the OBLIGATORINESS field, this does not mean that it is obligatory to query the TITLE attribute with both operators, but that it is obligatory to query it at least with one of them. In order to have the TITLE attribute appear obligatorily in the query with both operators (this is not possible in the form in the example), this should be done with two different 5-tuples for the TITLE attribute, one for each operator:



{(TITLE, {ANY}, OBL, 1, ANY) (TITLE, {=}, OBL, 1, ANY)}. Thus, as can be seen, when you want to differentiate the treatment of a specific attribute according to the operator with which it is used, more than one 5-tuple can exist for each attribute. When views or relations of the global schema are created, Virtual DataPort can automatically calculate its search methods from those of the sources and the expression used to define the view. This allows the system to know a priori whether or not a specific query can be responded to. Therefore, for derived views DataPort allows the query capacities of same to be consulted, but they cannot be modified directly.

5.6.6 View Configuration Properties

View Configuration Properties allow indicating specific characteristics of the underlying data sources such as their distributed transaction support capacity or whether insert operations are allowed. Section 5.5.14 indicated the configuration properties of the data sources. This section describes the configurable properties in each view, depending on the type of data source they have come from. NOTE: Typically, users do not need to edit this information since DataPort automatically uses suitable configurations for most common data sources. The properties of each base view can be configured by selecting the base view from the left-hand menu, either in the “DataSources” or in the “Combine” area, and pressing the “Advanced” button so that the advanced configuration window of the base view appears. Underneath the search methods (using the scroll bar where necessary) is the “Wrapper Source Configuration” link that leads to the property configuration screen. They can also be configured from VQL (see the Advanced VQL Guide [VQL] for more information). The configurable properties are as follows:

- Allow Insert: This indicates whether the data source underlying the view accepts inserting operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are:

o Default: VDP assigns a default value depending on the source type. In the case of relational sources and CUSTOM wrappers, the default value is “Yes”.

o Yes: the data source allows for inserting operations. o No: the data source does not allow for inserting operations.

- Allow Delete: This indicates whether the data source underlying the view accepts row deleting operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are:


o Yes: the data source allows for deleting operations. o No: the data source does not allow for deleting operations.

- Allow Update: This indicates whether the data source underlying the view accepts row updating operations. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are:


o Yes: the data source allows for updating operations. o No: the data source does not allow for updating operations.

- Delegate All Operators. This indicates whether the view allows for all operators to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate AND Condition. This indicates whether the view allows for the AND condition to be delegated. The value is “true” by default for base views from CUSTOM wrappers.

- Delegate Array Literal: This indicates whether the view allows for array-type compound constants to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.



- Delegate Compound Field Projection: This indicates whether the view allows projections on compound fields to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Left Function: This indicates whether the view allows for conditions with functions on the left part to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Left Literal: This indicates whether the view allows for conditions with constants on the left part to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate NOT Condition: This indicates whether the view allows for the NOT condition to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate OR Condition: This indicates whether the view allows for the OR condition to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Register Literal: This indicates whether the view allows for register-type compound constants to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Right Field: This indicates whether the view allows for conditions with fields in the right part to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Right Function: This indicates whether the view allows for conditions with functions on the right part to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Delegate Right Literal: This indicates whether the source allows for conditions with constants on the right part to be delegated. Applicable to CUSTOM wrappers. The value is “true” by default.

- Supports Distributed Transactions: This indicates whether the underlying data source can take part in an XA [XA] distributed transaction. It is applicable to relational databases (accessible via JDBC and ODBC) and CUSTOM wrappers. The possible values are:


o Yes: the data source meets the XA specification. o No: the data source does not meet the XA specification.

- Data in Order Field List: This indicates the fields by which the data is ordered. They are separated by commas, and the field names are accompanied by the ASC/DESC modifier. Property applicable in all data sources.

- Delegate Operators List: This property determines the list of operators that can be delegated to the data source. This allows for VDP to optimize the query plan created using the query made by the user, delegating part of the processing to the native source. While VDP carries out this action automatically on relational databases, other source types do not provide this information in their metadata, despite this sometimes being possible. VDP allows indicating the list of operators that can be delegated in Web Service and CUSTOM data sources (all operators can be delegated by default in CUSTOM wrappers). Operator "=" can be delegated by default in WS wrappers.

5.6.7 Configuration Properties for Specific View Types

The Denodo Aracne (see section 5.5.9) and the Google Enterprise Search / Mini (see section 5.5.8) data sources have specific configuration properties. The specific properties of these types of base views can be configured by selecting the base view from the left-hand menu, either in the “DataSources” or in the “Combine” area, and clicking on the “Advanced” button so that the advanced configuration window of the base view appears. Underneath the search methods (using the scroll bar, if necessary) is the “Wrapper Specific Configuration” link that leads to the property configuration screen. In the case of base views from Google-type sources, the configurable properties are: “Site Collections”, “Client”, “Languages” and “Number of Key match”. See section 5.5.8 for more information about these properties. In the case of base views from Denodo Aracne-type sources, the configuration properties are those relating to obtaining the most relevant terms of a document (or a field of it). See section 5.5.9 for more information about these properties.



5.7 STORED PROCEDURES

Virtual DataPort allows writing stored procedures using the JAVA language (see Advanced VQL Guide [VQL]). The DENODO_HOME/samples/vdp/storedProcedures path includes different example stored procedures. The README file in this path describes their function and contains the instructions to compile and install them. The stored procedure CalculateAvgRevenue, included in these examples, will be used in this section. Note: to follow this example you must:

1. Execute the script $DENODO_HOME/samples/vdp/storedProcedures/scripts/compile_storedprocedures.bat

2. Load the following Jar file using the Jar Management option: DENODO_HOME/samples/vdp/storedProcedures/target/jars/denodo-demo-storedprocedures.jar This option is located in the File – Extensions – Jar Management menu. See section 8.3 for more information.

5.7.1 Importing Stored Procedures

To import a new stored procedure, open the ‘Combine – Projects – ProjectName – Stored Procedures’ tab and click the option ‘New’. The new stored procedure dialog will be displayed (see Figure 77). The following data are requested on this screen:

• Name. Name of the new stored procedure. In our example, CalculateAvgRevenue. • Class name. Full name of the class that implements the stored procedure. In our example,

com.denodo.vdp.demo.storedprocedure.CalculateAvgRevenue. This class can be obtained from:

o A Jar file in server’s file system (click ‘Browse’ to select it) o A Jar loaded into the server with the ‘Jar management option’ (see section 8.3) In this case,

select the ‘Select Jars’ checkbox.

• Jar file. Optional. Path to the .jar file containing the class implementing the stored procedure. If the class implementing the stored procedure has not been added to the DataPort server, this option can be used to provide the path to the .jar file that contains it.

• Select Jars. If selected, select the Jar imported with the ‘Jar Management’ option that contains the class

indicated in the ‘Class name’ text box. Note: to select two or more Jars, hold the key Ctrl and click on the required Jars. This may be necessary if the stored procedure rely on external jars that have also been imported into DataPort.



Figure 77 Importing a Stored Procedure

On pressing ‘Ok’, the base view is created and the Tool displays a dialog with the properties of the new stored procedure and its input and output parameters (see Figure 78). The parameters of a stored procedure can be of types IN, OUT or IN_OUT. The IN parameters are the required input parameters to execute the procedure; the OUT parameters are the ones returned by the execution of the procedure; and the IN_OUT parameters are input parameters that are also returned by the procedure. The ‘Nullable’ column indicates that the value of that parameter can be NULL. Figure 78 shows information about the new stored procedure: It has an input parameter (IN) called taxid_list, of type array, and an output parameter avgrevenue (OUT), of type double.



Figure 78 Properties of a Stored Procedure

5.7.2 Executing Stored Procedures

To execute a stored procedure, press the ‘Execute’ button from the screen displaying the stored procedure properties. The tool will display a dialog similar to Figure 79 where the user has to fill in the input values; after clicking ‘Ok’, the procedure will be executed. Any query execution can be cancelled at any moment. If results are not being shown yet, cancellation can be achieved by pressing the ‘[X]’ button of the ‘Please Wait’ dialog which can be seen until the first batch of results is shown. Once the query results are being shown, the execution can be stopped by pressing the ‘Stop’ button. Figure 79 shows the screen in which the values of the input parameters for the procedure CalculateAvgRevenue are specified. This sample procedure receives as input a parameter of type array of registers called taxid_list. Each register has a field called taxid (client’s Tax ID). In our example, we create two registers with the values ‘B78596011’ and ‘B78596012’.



Figure 79 Execution of the Stored Procedure CalculateAvgRevenue

Figure 80 is the result of the execution.



Figure 80 Result of the execution of the stored procedure CalculateAvgRevenue

Stored procedures can be also invoked from the VQL Shell using the CALL statement (see “Use of Stored Procedures” in Advanced VQL Guide [VQL]).

5.7.3 Use of Stored Procedures in Creating Views

Virtual DataPort stored procedures can be used to create new derived views. More specifically, an invocation of a stored procedure may appear in the FROM clause of any query or view definition. The stored procedure will be considered a view which schema includes all input and/or output parameters (that is, of type IN, OUT, IN_OUT) of the stored procedure, and with a single search method (see section 5.6.5). The search method of a view generated from a stored procedure will have one obligatoriness (OBL) constraint for every input parameter (IN or IN_OUT). To create a view that uses a stored procedure in the FROM clause, ‘drag&drop’ the procedure on the creation dialog of the new view (union, join, projection/selection, etc) Then, if the procedure includes input parameters, they can be added to the new view by unselecting the ‘Use Input Parameters’ checkbox (see Figure 81). Otherwise, the user has to fill in the input values.



Figure 81 Parameters of a procedure added to a new view


Publication of Web Services 112

6 PUBLICATION OF WEB SERVICES

6.1 WEB SERVICE TYPES

Virtual DataPort can publish the content of one or several views (and/or stored procedures) as a Web service to provide access to the information of these views from any external application. It can also publish the result of any VQL statement. These Web services can be deployed in the Web services container embedded in the Denodo Platform or deployed in an external application server. The Web services can be published in the following versions:

• SOAP [SOAP] Web Services. • REST-style Web Services that use HTTP directly as the transport protocol and return data encoded in XML. • JSON Web Services. Similar to the REST-style Web services, although the output will be in JSON format

[JSON]. • RSS Web Services. Similar to the REST-style Web services, although the output will be produced in the

RSS format. • HTML Web Services. Similar to the REST-style Web services, but the output is an HTML table containing

the response data for the query executed. The table includes JavaScript code to sort the results by any field and/or paginate the returned results. It is also possible to adjust the size of the table and the cells and to modify its graphic appearance using a CSS file. See section 7 for more options to generate visual widgets from Virtual DataPort views and/or stores procedures.

The following section describes the publication process for these Web Services.

Figure 82 Publishing a Web Service



6.2 PUBLISHING WEB SERVICES

Views / Stored procedures / VQL statements are published using the ‘Publish – Web Services’ tab in the Administration tool. This tab displays the status of the Web Services container embedded in the Denodo Platform and the list of Web services currently created (see Figure 83). A service can be deleted by clicking on the Drop option alongside its name in the drop-down menu on the left of the screen. The following information is provided for each service:

• Name of the Web service. • Status. This indicates whether the service is deployed or not. If it is deployed but changes have been made

to the configuration of the service after deploying it, this is indicated by an “*”. • Context path. Root path to the service in the embedded application server. • Export. Contains links to generate the different artifacts associated with the Web service (.war file and/or

wsdl file). Both files generated will be accessible for downloading at the /export path of the Web container embedded in the Platform (e.g. http://localhost:9090/export).

o The .war file contains the implementation of the Web Service in the versions chosen during publishing. This is useful if you want to deploy the service in an external JEE Web container (such as Apache Tomcat [TOM]). Before exporting, you will be asked for the URI of the DataPort server and the user name and password to access it.

o The .wsdl file [WSDL] defines the interface of the SOAP version of the Web Service generated. It can be used with a utility for SOAP Web Services programming (such as those included in Apache Axis [AXIS]) to generate the necessary stubs to implement a client program accessing the SOAP Web service. For information on how to access the REST, JSON, RSS, and HTML versions of the Web Service see section 6.3.

• Deployment. If the service is not deployed in the Web container embedded in Virtual DataPort, the available option is ‘Deploy’. If it is already deployed, the options are ‘Redeploy’ or ‘Undeploy’.



Figure 83 Web services

The steps to generate a new Web Service are:

1. Click on the ‘new’ option of the “Web Services” tab (see Figure 82).

2. Select a view (or stored procedure) to export using the “Projects” drop-down menu on the left and drag&drop to the work area on the right of the screen. In our example, drag&drop the inc_grouped_by_pref_clients view created in section 5.4.4.

3. Alternatively, you can also click on the ‘Publish from vql expression’ button to add an operation that

will return the result of a VQL statement. This statement can include interpolation variables that will be considered the input parameters of the Web Service operation. See ‘Execution context of a query and interpolation strings’ in the Advanced VQL Guide [VQL] for more information about interpolation variables.

4. When adding the view or stored procedure, DataPort analyzes the query capabilities of it and

generates a series of “candidate operations” to query and/or update the view through the Web Service (Figure 84 shows the operations generated for our example). The generated operations have these characteristics:

• The input parameters of the generated operations correspond to the attributes of the view or

the input parameters of the stored procedure. • In our example, the generated operation is getINC_GROUPED_BY_PREF_CL, which

receives three input parameters that correspond to the attributes of the view: taxID, revenue and num_incidences.

• The return value of the operations generated will be an array of elements. Each of these elements will correspond to a tuple of the view and will have one subelement for each attribute.



• If the view published is updateable, default operations will also be generated to insert, update and delete tuples from the view (see section 6.2.1 for further details).

Figure 84 Operations created for the view inc_grouped_by_pref_clients

5. It is possible to modify operations as follows: • Delete an operation: click on the ‘Drop’ option alongside its name. • Change the operation’s name: clicking on the ‘Rename option’. • Delete an input parameter of an operation: click on the Drop option alongside the name of the

parameter). Depending on the query capabilities of the view (see section 5.6.5), some operations generated may include mandatory parameters that cannot be deleted.

• Delete a return parameter. • Indicate the operator used to query DataPort with the value provided for this parameter: click

on the current operator and select the new one.

6. A Web Service can include operations based on more than one view. To do so, simply drag & drop from the new view and repeat steps 2 to 5.

7. Change the name of the new Web Service: fill in the ‘Edit Web Service Name’ text box and click on

the arrow button.

8. Click on the ‘Advanced’ button to configure additional settings of the Web Service publication:

• Access types to be generated for the service: SOAP, REST, JSON, RSS, and/or HTML.

• For SOAP Web services only, the Web Service style: RPC or DOCUMENT. Some Web Service consumer applications may require a specific style.



• For SOAP, REST and/or JSON Web services, the type of data to be used for exporting the date-type fields. The available options are the XML Schema date and dateTime data types. The XML Schema date type only considers the day, month and year, whereas the dateTime includes full information on time, including hours, minutes, seconds, and milliseconds.

• SOAP XSLT Transformations. We can transform the SOAP requests and responses to adapt

them to the requirements of an existing SOAP client. See section 6.2.3.

• REST XSLT Transformations. We can transform the REST responses to adapt them to the requirements of an existing client. See section 6.2.4.

• SOAP over JMS. Virtual DataPort can subscribe to a JMS server to listen to SOAP messages.

See section 6.2.5.

• CSS File. For Web services with HTML output, it is possible to specify a CSS file that adapts the graphic appearance of the HTML output generated by DataPort. By accessing this option, it is possible to view the CSS document to be used and load a new one, if required.

• RSS Mappings. The RSS format imposes a series of specific fields for each item of its output.

Therefore, on exporting a view in RSS format, the correspondence between the fields of the view and the fields in RSS format must be specified. See section 6.2.6 for further details.

• Configuration of the authentication of the new Web Service. See section 6.2.7.

• Configuration parameters for connections. The Web services published will be deployed to an

application server (usually in the Web Service container embedded in the Denodo Platform). Each time a Web service operation is invoked, it will produce a statement executed against the DataPort server to obtain the required results. It is possible to configure certain connection parameters to be used by the Web service to access DataPort (see section 6.2.2 for further details).

9. Click on ‘Ok’ to create the Web service. To deploy it, click on ‘Deploy’ in the table of existing Web

Services. After creating a Web service, you can open it by clicking on its name in the ‘Publish’ tab. Then, you can:

• Add more views/stored procedures to publish. • Edit the existing operations of the Web service

• Change the Advanced settings of the Web service.

• View the VQL code that generates the Web service by clicking on View VQL. This dialog has the same

options as the View VQL dialog of views (see section 5.3.4.1), except Show dependencies. This dialog does not have this option because it only displays the VQL code of the Web service.

6.2.1 Operations for Updatable Views

If the published view is updateable (see “Inserts, Updates, and Deletions” section in the Advanced VQL Guide [VQL]), the Tool will generate operations to insert, update, and delete tuples from that view. More specifically, the following operations will be generated:



• An insert operation with a parameter for each field of the view. The name of the operation will start with “insert” followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called insertINTERNET_INC).

• A delete operation with a parameter for each field of the view. The name of the operation will start with

“delete” followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called deleteINTERNET_INC).

• An update operation with two parameters for each field of the view: one of the parameters will take the

name of the field and the other will take the name of the field prefixed by “New”. The parameters with the prefix “New” correspond to the new values (SET clause of the UPDATE statement). Whereas those without this prefix are used to choose the tuples to be updated (WHERE clause of the UPDATE statement). The name of the operation will start with “update” followed by the name of the view (e.g. if the view is called INTERNET_INC, the operation will be called updateINTERNET_INC).

IMPORTANT NOTE: On including insert / update /delete operations in the services published, you are enabling external clients to insert, delete or update tuples. If you want to avoid this, use the Drop option alongside each operation to delete them from the Web service published.

6.2.2 Configuration of Connections in the Web Services Published

The Web services exported will be installed in an application server (typically in the Web service container embedded in the Denodo Platform). Each time a Web service operation is invoked, it will execute a statement against the DataPort server to obtain the required results. It is possible to configure certain connection parameters to be used by the Web service to access DataPort. The most important aspect that can be configured is the use of a connection pool. Hence, the connections opened by the Web service with DataPort may be reused. In production environments, the use of the connections pool is strongly recommended. Connection parameters are configured in the “Connection Parameters” section of the advanced configuration of the publication process of a new Web service. The parameters that can be modified are:

• Chunk Size, Chunk Timeout, and Query Timeout. Their interpretation is the same as in any other VDP client (see VDP Developer’s Guide [DEV]).

• Enable Pool. This box must be marked to enable use of the connection pool. • Initial Size. Initial number of connections to be opened in the pool. • Max Active. Maximum number of connections in the pool. A negative value means there is no limit.

6.2.3 SOAP XSLT Transformations

In an environment with existing SOAP clients and services, we do not need to modify those clients to work with Virtual DataPort Web services. We can define XSLT stylesheets [XSLT] to transform the incoming SOAP messages to adapt them to the format that the new Web service expects. We also can define stylesheets to transform the SOAP responses before sending them to the existing clients. Follow these steps to define XSLT transformations for a Web service:

1. Open the “Advanced” configuration of the Web service. 2. Click on “SOAP XSLT Transformations”.



3. Select an operation and click on “Add Transformation” 4. Select the “Input XSLT Transformation” box and write the stylesheet. This stylesheet will be applied to the

incoming SOAP messages to transform them into the format expected by the Virtual DataPort Web service.

5. Select the “Output XSLT Transformation” box and write the stylesheet. This stylesheet will be applied to the SOAP responses generated by the Web service to transform them into the format expected by the existing SOAP client.

6. If the existing SOAP client sends the parameter “SOAPAction” with its requests and the parameter cannot

be changed, select the “Incoming SOAPAction” box and write it.

7. Repeat steps 3 to 6 for each operation of the Web service that we want to transform. See appendix 12.7 for more details on how to generate these XSLT stylesheets.

6.2.4 REST XSLT Transformations

In an environment with existing REST clients, we do not need to modify those clients to work with Virtual DataPort Web services. We can define XSLT stylesheets [XSLT] to transform the XML responses of the Virtual DataPort REST services, before sending them to existing clients. Follow these steps to define XSLT transformations for a Web service:

1. Open the “Advanced” configuration of the Web service. 2. Click on “REST XSLT Transformations”. 3. Select an operation and click on “Add Transformation”.

4. Select the “Output XSLT Transformation” box and write the stylesheet. This stylesheet will be applied to

the REST responses (XML) generated by the Web service to transform them into the format expected by the existing client.

Repeat steps 3 and 4 for each operation of the Web service that we want to transform.

6.2.5 SOAP over JMS

SOAP is transport-independent and can be bound to any protocol. Although it is usually used with HTTP, it can also be used with JMS (Java Message Service [JMS]). When using SOAP over JMS [SOAP_JMS], the client sends the SOAP message to the JMS server, which forwards it to the Web service. Then, the Web service sends the response back to the JMS server, which forwards it to the client. SOAP over HTTP is more interoperable as there is more support for it. However, there are certain factors that we can only achieve using SOAP over JMS:

a) Scalability. By using SOAP over JMS, clients can send requests without waiting for the response. That way, the server does not have to process all the requests as they arrive and clients are not blocked waiting for a response.

b) Reliability. The JMS server ensures that requests and responses are delivered. In case of any failure in

the communication, the JMS server keeps trying to send the messages. This is important in environments that deal with critical data.



Virtual DataPort can subscribe to a JMS server to listen to SOAP messages. Follow these steps to enable this feature:

1. Open the “Advanced” configuration of the Web service. 2. Click on “SOAP over JMS”

3. Select “On” to enable SOAP over JMS support 4. “Destination” is the name of the queue or topic that Virtual DataPort will subscribe to, waiting for SOAP

messages. Depending on the vendor of the JMS server, we might have to create the destination, or it is created automatically when the new Web service tries to subscribe to it.

5. Select “Queue” or “Topic”.

6. “User name” and “Password”. Credentials to connect to the JMS server.

7. “JMS vendor”. Select GENERIC if the vendor of the JMS server is not in the list and it can be accessed via

JNDI. In this case, we have to provide the appropriate JNDI connection properties (see appendix 12.8).

8. Click ‘Ok’. When we deploy the Web service, it connects to the selected JMS server and subscribes to the JMS queue or topic, so it can receive the messages sent to that destination. IMPORTANT NOTE: a Web service with SOAP over JMS enabled needs some client jars to be able to connect to the JMS server. The vendor provides these jars. Appendix 12.8 contains lists of required client jars for the most popular vendors.

• The first time a Web service is deployed into the embedded Web container of Virtual DataPort, copy these jars to the directory $DENODO_HOME/resources/apache-tomcat/common/lib. Then, restart the Virtual DataPort server so it can load the Java classes of these jars.

• If the Web service is deployed into an external application server, we have to copy these jars into the

/WEB-INF/lib directory of the war file, before deploying it.

6.2.6 Mappings in the Publication as RSS

The RSS 2.0 format [RSS] specifies a series of specific fields for each item. Therefore, on exporting a view in RSS format, the correspondence between the fields of the view and the fields in RSS format must be specified. The following options are displayed on the RSS mappings screen:

• If the service published includes operations of more than one view (or stored procedure), the RSS version of the service may only include the operations to query one of them. The selectable at the top left enables you to choose the view or stored procedure to be used.

• Values for channel attributes. An RSS feed contains a channel element that specifies general information

on the feed. In this group of options, it is possible to specify constant values for each of the channel subelements permitted by RSS format.

• Mappings for the attributes of channel items. An RSS feed contains a list of item elements. DataPort will

generate an item element for each row returned by the query executed on the view or stored procedure selected. This group of options enables you to select the attribute of the view that corresponds to each



item subelement defined in the RSS format. If the mapping for a certain item subelement is left with the ‘none’ value, that subelement will not be included in the output feed. The RSS format specifies that at least one value must be assigned either to the ‘title’ subelement or to the ‘description’ subelement.

6.2.7 Web Services Authentication

The available authentication options depend on the service type: For REST, JSON, HTML and RSS the options are:

• None. The Service can be accessed without authentication.

• HTTP Basic. Basic HTTP authentication [HTTP-AUTH] with the credentials passed as plain text. All the users will use the values of the fields ‘Login’ and ‘Password’ as credentials for accessing this Service.

• HTTP Basic with VDP. Basic HTTP authentication using the credentials of the Virtual DataPort’s users.

That is, the Web service will connect to Virtual DataPort with the credentials used by the client of the Web service. Only users whose user name is in the ‘Accepted user(s)’ list will have access to the Service (separate user names with commas). If this list is empty, the Web Service will accept all Virtual DataPort’s users. Unlike with the other authentication methods, with this one, we have to grant the user privileges to access the published views.

• HTTP Basic with LDAP. Basic HTTP authentication using an LDAP server. To configure this

authentication follow these instructions: 1. Select the LDAP data source representing the LDAP server used for authenticating users. See section

5.5.10 to learn how to create an LDAP data source.

2. Fill in the ‘User pattern’ text box. The ‘User pattern’ is used to build the user's Distinguished Name replacing the @login token with the received user name. I.e. if the User pattern is cn=@login,ou=People,dc=YourOrganization,dc=com, the @login token will be replaced by the user name provided by the invoker of the Service.

3. Accepted user(s): only users whose user name is in this list will have access to the Service. To accept

more than one user, separate their user names by commas. If empty, every user authenticated by the LDAP Server will be granted access to the Service.

• HTTP Digest. HTTP Digest access authentication [HTTP-AUTH]

The authentication options for SOAP Web Services are the same as for REST, JSON, HTML and RSS, but also support the Web Services Security protocol [WSS]

6.2.8 Types Conversion Table for REST / SOAP Published Web Services

Table 1 shows the conversions applied between the DataPort data types and the data types used for the input and output parameters of the Web services generated in the SOAP and REST versions. In this table, the xsd prefix identifies the XML Schema types and the soapenc prefix, the SOAP types.

DataPort Data Type SOAP Web Service Data Type REST Web Service Data Type text xsd:string xsd:string long xsd:long xsd:long double xsd:double xsd:double



int xsd:int xsd:int boolean xsd:boolean xsd:boolean link xsd:string xsd:string float xsd:float xsd:float money xsd:double xsd:double time xsd:int xsd:int date xsd:date or xsd:dateTime xsd:date or xsd:dateTime blob xsd:hexBinary xsd:hexBinary xml xsd:string xsd:string enumerated xsd:string xsd:string array xsd:complexType

(restriction on soapenc:array)

xsd:complexType (sequence)

register xsd:complexType (sequence)

xsd:complexType (sequence)

Table 1 Conversions between DataPort data types and Web service parameter types

6.3 INVOKING THE EXPORTED WEB SERVICES

This section describes how to invoke the different types of Web services. The ‘Context Path’ column of the list of published Web services contains the path to each Service. This path contains a list of the exported versions for that service. The relative paths /services, /rest, /json, /html and /rss show the available operations for each Web service type. Note: In the following examples, we will suppose that the embedded Web Container is running on port 9090 of the acme host and the name of the exported web service is testWS of the admin database. Example: the following table contains a list of the URLs of the information pages: WS Version URL

SOAP http://acme:9090/server/admin/testWS/services

REST http://acme:9090/server/admin/testWS/rest

JSON http://acme:9090/server/admin/testWS/json

HTML http://acme:9090/server/admin/testWS/html

RSS http://acme:9090/server/admin/testWS/rss

Table 2 URLs of Web Service’s information pages

These information pages list the operations of the Web Service. The SOAP information page contains a link to the wsdl file of the service. The REST information page contains a link to the XML Schema of the XML output of each operation. The links to the XML Schemas of the REST services’ output have this format: http://host:port/server/databaseName/serviceName/rest/opName/xsd



Example: If the testWS Web service has an operation called getINTERNET_INC, the URL of the XML Schema of its response is:

http://acme:9090/server/admin/testWS/rest/getINTERNET_INC/xsd

Table 3 Obtaining the XSD Schema of a REST operation

The URL to invoke an operation of a REST service has this format: http://host:port/server/databaseName/serviceName/rest/opName? paramName1=value1&...&paramNameN=valueN , where N is the number of parameters of the operation. The format for the HTML, JSON and RSS services is the same but replacing ‘rest’ by, respectively, ‘html’, ‘json’ and ‘rss’. Note: SOAP Web Services should not be used invoking an URL, but using a SOAP client instead. Example: Let us suppose that the testWS service has an operation called getINTERNET_INC that requires no parameters. The operation can be invoked as follows: WS Version URL

REST http://acme:9090/server/admin/testWS/rest/getINTERNET_INC

JSON http://acme:9090/server/admin/testWS/json/getINTERNET_INC

HTML http://acme:9090/server/admin/testWS/html/getINTERNET_INC

RSS http://acme:9090/server/admin/testWS/rss/getINTERNET_INC

Table 4 Invoking a Web Service without parameters

Example: Let us suppose that the testWS service has an operation called getINTERNET_INCBYIINCID that requires one input parameter called iinc_id. The operation can be invoked as follows: WS Version URL

REST http://acme:9090/server/admin/testWS/rest/ getINTERNET_INC?iinc_id=1

JSON http://acme:9090/server/admin/testWS/json/ getINTERNET_INC?iinc_id=1

HTML http://acme:9090/server/admin/testWS/html/ getINTERNET_INC?iinc_id=1

RSS http://acme:9090/server/admin/testWS/rss/ getINTERNET_INC?iinc_id=1

Table 5 Invoking a Web Service with one parameter

If some parameter is of compound-type, its value will be represented by using the ROW and {} VQL constructors (see section “Conditions with compound values” in VQL Advanced Guide [VQL]). Example: Let us suppose that the testWS Web service has an operation called getREVENUESUM. This operation publishes the view REVENUESUM created in section 5.4.3.1. It has a record array-type input parameter



called clients. Each record has one text-type field that represents a company’s tax identifier. The operation returns the sum of the revenue of each company in the input parameter. The URL to invoke the REST version of this operation is:

http://acme:9090/server/admin/testWS/rest/getREVENUESUM? clients={ROW('B78596011'),ROW('B78596012')}

Table 6 Invoking a REST Web Service with an array-type parameter

By default, the REST, JSON and HTML Web Service return an error if they receive parameters that do not belong to the published view. If the Web Service client needs to ignore extra parameters, it must add the validateparams parameter to the URL. For example, if a client invokes the URL

http://acme:9090/server/admin/testWS/rest/ getINTERNET_INC/?made_up_parameter=1

the Service returns an error because ’made_up_parameter’ does not belong to the published view. If the client invokes

http://acme:9090/server/admin/testWS/json/ getINTERNET_INC/validateparams/false?made_up_parameter=1

the Service will ignore ‘made_up_parameter’ and return the result of the query. The param validateparams has to be added to the URL after the name of the operation and is valid for the REST, JSON and HTML Web services.

6.3.1 HTML Output Configuration

The HTML version of the Web services can be invoked with additional parameters in order to configure the HTML table that displays the results of the queries:

• shownumresults. If true, the table will also display the number of rows obtained by the query.

• intervalsize. If present, the results of the query will be paginated. The value of the parameter is the number of results in each page.

• maxresults. Maximum number of results to be displayed. If the query returns more rows, all excess

results will be omitted.

• cellwidth. Maximum cell width expressed in number of characters. If the text of a cell is wider, the text will be divided in several lines.

• cellheight. Maximum number of lines in a cell after having splitted the text according to the

cellwidth parameter. If this is exceeded, all the cells of this column will have a scroll bar.

• width. Maximum width (in pixels) of the table. If the size is exceeded, a scroll bar is added.

• height. Maximum height (in pixels) of the table. If the size is exceeded, a scroll bar is added.



• noescapehtml. List of the names of the columns which HTML code will not be escaped (separate each name with comma). By default, the HTML of all the cells is escaped, unless its column name is in this list.

These parameters must be indicated in the part of the URL corresponding to the access path (before the query parameters) in the following format: http://host:port/server/database/serviceName/rest/opName/ paramName1/value1/.../paramNamen/valuen For example, the following expression invokes the getINTERNET_INC operation, limiting the number of results displayed to 50 and setting the number of rows per page to 10. http://acme:9090/server/admin/testWS/html/getINTERNET_INC/ maxresults/50/intervalsize/10/?iinc_id=1/

Table 7 Invoking the HTML Web Service with configuration parameters

6.3.2 JSON Output Configuration

The JSON version of the Web service can return JSON data, prefixed with the name of a function (also called JSON with padding or JSONP). That way, when a browser receives the response, it receives a script rather than data. To do it, add the parameter jsoncallback to the parameters of the URL. E.g: http://acme:9090/server/admin/testWS/json/getINTERNET_INC? jsoncallback=js_function

Table 8 Invoking the JSON Web Service with padding (JSONP)

This will return the following: js_function( <result of the query> ); Note: to use this feature, the published view cannot have a field called ‘jsoncallback’.


Publication of Views as Widgets 125

7 PUBLICATION OF VIEWS AS WIDGETS

Virtual DataPort allows a view or stored procedure to be published as a visual widget. Then, these widgets can be exported to different widget technologies:

• Java Portal Servers (Portlets JSR-168 [JSR-168] and JSR-286 [JSR-286]) • Microsoft SharePoint Web Parts [SHAREPOINT] • OpenAjax [OPENAJAX]

These widgets allow querying and displaying the data of the selected view and can interact with other widgets sending them the data obtained from the view or receiving it to filter the contents of the view that are shown.

7.1 PUBLISH A VIEW AS A WIDGET

Widgets are created and published from the ‘Publish – Widgets’ tab of the Administration Tool. When accessing this tab, we can see a table containing the list of all the existing widgets. To create a new widget, click on the ‘New’ button of the ‘Publish – Widgets’ tab. The following information must be provided:

• Widget name. Name of the new widget.

• Display name. Name that will be displayed to the users.

• Element to publish. Select the view/stored procedure to publish by selecting it in this drop-down list. Alternatively, it is possible to:

o Drag & Drop or click on the element in the Projects list in the left panel. o Use the Quick Search.

• Target project. Project where the new widget will be stored.

• Configure the Advanced settings of the widget:

o Enable Help Mode. The three widget technologies have a Help Mode that, if enabled, displays

information about how to use the widget. There is a default help text that can be changed by clicking on Edit help mode content.

o Custom Table Events (only for JSR-286 portlets). Select this option to enable the Full Table JSR-286 complex event. Note: increase the performance of the JSR-286 portlets by deselecting this option (see 7.3 for more detail).

o Connection parameters. Configure the parameters used by the portlets and the auxiliary Web

Service to connect to Virtual DataPort (see section 0 to determine the auxiliary Web Service of the widget) It is usually correct to use the default values for these parameters.

Chunk Size. Maximum number of results that a block can contain. If this parameter is 0 or empty, all the results will be returned in a single block.

Chunk Timeout. Maximum time (in milliseconds) the server will wait before returning a new block. After that time, the current block will be returned even if it does not contain the number of results specified by the Chunk Size parameter.



If this parameter is 0 or empty, all the results will be returned in a single block whenever the query execution ends.

Query Timeout. Maximum time (in milliseconds) the server will wait for the termination of a query. If its value is 0, it will wait indefinitely until the query finishes.

Enable Pool. Enable the use of the connection pool. In production environments, the use of the connections pool is strongly recommended.

Initial Size. Initial number of connections to be opened in the connection pool. Max Active. Maximum number of connections in the pool. A negative value means

there is no limit.

• Click ‘Ok’ and click on the ‘Publish – Widgets’ tab to open the Widgets Status Table (see Figure 85)

Figure 85 Widgets Status Table

This table provides the following information for each widget:

• Name of the widget. • Export. Contains links to export the widget to different widget technologies.

• Status indicates whether the auxiliary Web Service is deployed in the embedded Web container. If it is

deployed, but changes have been made to the widget configuration, this is indicated by an “*”.

• Context path. If the auxiliary Web Service is deployed in the embedded Web container, contains the path to the service.

• Deployment. Contains links to deploy / redeploy / undeploy the auxiliary Web Service of the widget.

• Properties. Click on ‘Show’ to display the properties of the widget.



7.2 AUXILIARY WEB SERVICES

The Open Ajax Widgets and the Microsoft Web Parts cannot directly connect to the Virtual DataPort server, as the Java Portlets do. Therefore, they require an auxiliary Web Service that will retrieve the data from DataPort and send it back to these widgets. Every time a widget is created, its auxiliary Web Service is also created, but they still need to be deployed. There are two options to deploy an auxiliary Web Service:

1. Deploy it in the embedded container: click ‘Deploy’ in the ‘Deployment’ column of the Widgets Status panel.

2. Export it to an external J2EE Web Container (such as Apache Tomcat [TOM]): click ‘war’ in the

‘Deployment’ column of the Widgets Status panel and follow the required steps to deploy a web application in your specific environment.

Note: It is required to deploy the auxiliary Web Service before using an OpenAjax Widget or a Microsoft Web Part. Otherwise, these widgets will not display any data.

7.3 EXPORT TO JSR-168 OR JSR-286 PORTLET

A widget can be exported to a JSR-168 or JSR-286 portlet, so it can then be deployed it in any standard Java Portal Server. When clicking the ‘JSR-168’ or the ‘JSR-286’ link in the ‘Export’ column of the Widgets Status Table, the user will have to fill in:

• URI of the DataPort server that the portlet will obtain the data from. • Credentials to access to it.

This will generate a ‘.war’ file ready to be deployed in any standard Java Portal Server. This file can be obtained from:

• $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.war

• Or, http://localhost:9090/export/<name of the widget>.war This portlet will display the contents of the published view or stored procedure. In the EDIT mode, the portlet displays a query form where you can configure (see Figure 87):

• Query parameters that will be used to filter the contents of the exported view. • Hide some of the columns of the results table. • Results table settings: rows per page, height and width of the results table and its cells, and columns in

which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped) The main difference between the two types of portlets is that the JSR-168 version can only display the content of a view. On the other hand, the JSR-286 version can interact with other portlets by sending/receiving events or sharing public render parameters (see [JSR-286] to learn more about these terms) The JSR-286 portlet can interact with other portlets by:

- Sending simple events. When the user selects a row of the table by clicking the icon of the first cell of the row (see Figure 86), the portlet sends a simple event for every cell of that row. The payload of each event is the content of a cell.



- Sending row events. When the user selects a row of the table, the portlet sends an event containing an object of the type com.denodo.vdb.wsgenerator.common.execution.portlet.vo.DataRow

This object contains the data of the selected row.

- Sending table events. When the user clicks on the icon in the first cell of the header, the portlet sends an event containing an object of the type com.denodo.vdb.wsgenerator.common.execution.portlet.vo.DataTable This object contains all the data of the results table. To be able to develop a portlet capable of receiving the DataRow and the DataTable events, you need to add a reference to the library $DENODO_HOME/lib/portlet-client-core/portlet-client-core.jar

- Receive an event. The JSR-286 portlet can receive an event for every parameter of the query form. The

name of these events follows the convention ‘Input_’<name of the search parameter> - Sharing public render parameters. There is one render parameter for each column of the results table and a

render parameter for each parameter of the search form.



Click to send an event with all the data of the table (DataTable)

Click to send an event with all the data of the row and 3 events containing the value of each cell (DataRow)

Figure 86 JSR286 Portlet



Figure 87 Portlet search form

7.4 EXPORT TO OPENAJAX WIDGET

A Widget can be exported as an OpenAjax widget and then deployed in an OpenAjax compliant container such as the OpenAjax Mashup Editor [OPENAJAX]. When clicking the ‘OpenAjax’ link in the ‘Export’ column of the Widgets Status Table, the user will have to fill in:

• URL of the auxiliary Web Service of this widget. The default value of this field is the path of the auxiliary Web container, embedded in the DataPort server. So, if the auxiliary Web Service will be deployed to the embedded container, the default value is correct.

• Deployment URL: URL of the directory where the OpenAjax Widget will be located in the server where the OpenAjax container is deployed.

This will generate a ‘.zip’ file ready to be deployed in the OpenAjax container. This file can be obtained from:



• $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.zip

• Or, http://localhost:9090/export/<name of the widget>.zip Note: Remember to deploy the auxiliary Web Service.

7.5 DEPLOYMENT OF AN OPENAJAX WIDGET

Before deploying an OpenAjax widget, you must install an OpenAjax container (see appendix 12.5 for instructions on how to install the OpenAjax Mashup Editor). In this section, it is explained how to deploy the widget in the OpenAjax Mashup Editor (consult your container documentation if you are using another OpenAjax container): Follow these steps to deploy a new OpenAjax widget:

1. Copy the .zip file generated in the previous section to the $MASHUP_EDITOR_HOME/gadgets/samples/gadgets

2. Unzip the file into a new directory with the same name as the widget. I.e.: if the widget is called

‘internet_inc’, the name of the new directory has to be ‘internet_inc’. 3. Open the Repository Browser:

http://<hostname>/mashupapp/repository/scaffold.php 4. Click on the ‘Add’ button at the bottom of the screen and a dialog like the one in Figure 88 will be

displayed. 5. Click ‘Browse’ and select the OpenAjax metadata file located in

$MASHUP_EDITOR_HOME/gadgets/samples/gadgets/<name of the gadget>/<name of the gadget>_oam.xml

6. Fill in the ‘Version’ text box (the Version number is not important) 7. Click ‘Save’.



Figure 88 Adding a new item to the OpenAjax Repository

To open the OpenAjax Mashup Editor, follow these steps: 1. Open the URL

http://<hostname>/mashupapp/gadgets/samples/newmashup.php

2. Click on the down-arrow on the left upper side of the Mashup Tool, browse through the menu to locate the new widget and click on its name (see Figure 89)



Figure 89 Adding the new widget to the OpenAjax Mashup Editor

Now, the new widget is added to the Mashup (see Figure 90)



Figure 90 OpenAjax Widget displaying the content of the ‘incidents_sales’ view.

To open the widget’s search form, click on the down arrow on the upper left side of the widget and click on ‘Edit widget preferences’ (see Figure 91) In the search form (see Figure 92) you can configure:

• Query parameters that will be used to filter the contents of the exported view. • Hide some of the columns of the results table. • Results table settings: rows per page, height and width of the results table and its cells, and columns

which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped) Once you have finished, click ‘Search’ to go back to the results table.



Click on this icon to open the widget’s menu

Figure 91 Opening the OpenAjax widget configuration form



Figure 92 OpenAjax widget search form

To interact with other widgets you need to open the ‘Edit Widget Properties’ dialog (see Figure 93). To do this, click on the down arrow on the upper left side of the widget and click on ‘Edit Widget Properties’. In this dialog, you can bind the properties of the search form (properties that start with “Input_”) with properties of other widgets. This is useful if you want to filter the contents of the published view by the values received from another widget. You can also bind the value of the cells of the selected row with another widget. In Figure 94, the user is binding the property “SUMMARY” with the property “message” of another widget. After doing this, when the user selects a row (clicks on the icon of the first column of a row), the value of the cell SUMMARY for that row will be sent to the other widget.



Click to bind this property with another widget’s property. The received value will be used to filter the contents of the results table.

This property contains the value of the parameter ‘TAXID’ of the search form.

This property contains the value of the column ‘SUMMARY’ of the selected row.

Click to bind this property with the property of another widget.

Figure 93 ‘Edit Widget Properties’ dialog: binding properties of a widget with properties of other widgets



Figure 94 Binding the property of a widget with the property of another widget.

7.6 EXPORT AS MICROSOFT WEB PART

A widget can be exported as a Microsoft Web Part and then deployed in a Microsoft Office SharePoint Server [SHAREPOINT]. When clicking the ‘Web Part’ link in the ‘Export’ column of the Widgets Status Table, you will have to fill in the URL of the auxiliary Web Service of this widget. The default value of this field is the path of the auxiliary Web container, embedded in the DataPort server. So, if the auxiliary Web Service will be deployed to the embedded container, the default value is correct. After clicking ‘Ok’, DataPort will generate a ‘.zip’ file containing a “.webpart” file and an “.xml” file that contain the necessary information for displaying the view. The new file can be obtained from:

• $DENODO_HOME/resources/apache-tomcat/webapps/export/<name of the widget>.zip

• Or, http://localhost:9090/export/<name of the widget>.zip



7.7 DEPLOYMENT OF A MICROSOFT WEB PART

The first time you want to deploy a Web Part you have to deploy the Denodo SharePoint Solution located in $DENODO_HOME/webapps/webpart-core/DenodoWebPart-4.6.0.wsp. See appendix 12.6. After this, we are ready to deploy the exported Web Parts. To do this, follow these steps:

1. Unzip the exported “.zip” file in your SharePoint server and copy the “.xml” file to: C:\Inetpub\wwwroot\wss\VirtualDirectories\80\wpresources\DenodoWebPart

2. Open the “Web Part gallery“, usually located in http://localhost/_catalogs/wp/Forms/AllItems.aspx and load the “.webpart” file into the gallery (see Figure 95)

Figure 95 Loading the “.webpart” file into the Web Part gallery.

Now, the Web Part is ready to be added to any page As well as portlets and OpenAjax widgets, the Web Parts also have a query form to configure the following parameters (see Figure 96):

• Query parameters that will be used to filter the contents of the exported view. • Hide some of the columns of the results table. • Results table settings: rows per page, height and width of the results table and its cells, and columns

which HTML code will not be escaped (if a cell of a selected column contains HTML, it will not be escaped)

http://localhost/_catalogs/wp/Forms/AllItems.aspx



Figure 96 Web Part search form


Server Administration 141

8 SERVER ADMINISTRATION

This section describes the main options available to administer the Denodo Virtual DataPort server. Firstly, details are given about how to configure various aspects of the Virtual DataPort server. Then, how to export the Server metadata for backup and migration purposes is described.

8.1 CONFIGURING THE SERVER

To manage the databases, users and access rights of a Virtual DataPort server it is necessary to access the administration tool in server administration mode, which implies doing it with an administrator-type user. When accessing the server for the first time, the default administrator-type user can be used (user login “admin” with password “admin”). The server configuration parameters are explained in the following sections:

• Server connectivity: section 8.1.1. • Pool of threads: section 8.1.2.

• Cache: section 8.1.3.

• Limitation of concurrent requests (query queueing): section 8.1.4.

• HTTP proxy: section 8.1.5.

• Swapping parameters: section 8.1.6.

• Execution parameters of stored procedures: section 8.1.7.

• Internationalization configuration: section 8.1.8.

• JMS listeners: section 8.1.9.

8.1.1 Server Connectivity

Open the ‘Server Configuration – Server Ports’ dialog (see Figure 97) to configure several connectivity settings of the Virtual DataPort server:

• Port numbers of the Virtual DataPort server: execution, shutdown and auxiliary ports. The ‘Server Port Number’ is the port where the server listens to connections from the Administration Tool, JDBC clients and clients using the Virtual DataPort API. The ‘ODBC Port Number’ is the port where the server listens to ODBC connections.

• Port numbers of the Web container embedded in the Denodo Platform.

Important note: when the Virtual DataPort Server and the Administration Tool or other clients are installed in different machines, you may need to change the interface that the Server listens to connections. To change this, open the Denodo Control Center, open the ‘Configuration’ dialog and change the ‘RMI Host’ of the ‘JVM Options’. See more details in the section ‘Virtual Machine and Web Container configuration’ of the Installation Guide [DENINST]. The reason for this is that in some environments, when listening to ‘localhost’, the server does not allow connections from remote hosts.



Note: When the connection between clients and the Virtual DataPort server is established through a firewall, this must be configured to allow the connections to the execution and auxiliary ports.

Figure 97 Port Configuration

The ports changes will take effect the next time the Virtual DataPort server is launched.

8.1.2 Threads Pool

As mentioned earlier, whenever possible, the Virtual DataPort query execution engine operates concurrently on the data sources. Implementation of the concurrence is carried out through a multithreading subsystem that can be configured through the Virtual DataPort administration tool. To configure the multithreading subsystem of a Virtual DataPort server, access “Server Configuration - Threads Pool”. The configurable parameters are as follows:

• Max Threads: Maximum number of threads the pool can support simultaneously (default value is 300).

• Expiration Time: Maximum time an unused thread remains in the pool (default value is 600,000 milliseconds).

• Sleep Time: Indicates how often the threads not used in the pool are checked to eliminate those that exceed the Expiration Time (default value is 300,000 milliseconds).



• Timeout: Maximum time allowed by the pool for execution of one of the threads (default value is 600,000 milliseconds).

• Checkout Time: Maximum time the pool waits for a free thread. If this time is exceeded, the system displays an error message (30,000 milliseconds by default).

The configuration changes will become immediately effective.

8.1.3 Configuring the Cache

As mentioned earlier, Virtual DataPort incorporates a system (called cache module) to store local copies of the data retrieved from the data sources, in a JDBC database. Virtual DataPort embeds an Apache Derby [DERBY] database that can be used to store the cache data, but it is also possible to store the cache in the following external DBMSs:

• Oracle 8i, 9i, 10g and 11g. • MySQL 4.1 and 5.x (their drivers are not included).

• SQL Server 2000, 2005, 2008 and 2008R2.

To activate the cache, access the “Server Configuration – Cache” tab (see Figure 98), select the option “Cache status on” and fill in the following fields:

• DB adapter: Type of the database management system (DBMS) which will be used as cache. Virtual DataPort embeds an Apache Derby [DERBY] database which can be used to store the cache data by selecting the ‘Embedded Derby Server’ check box. In this case, it is not required to fill in any other field. This embedded database can be configured to be accessible remotely by clicking on the ‘Embedded Derby server configuration’ link (see below for more detail).

• Driver jar: Path to the .jar file of the JDBC driver.

Alternatively, you can copy the .jar file to the directory $DENODO_HOME/extensions/thirdparty/lib and leave this field empty. After copying the file, you have to restart the Virtual DataPort Server and the Administration Tool so the driver can be loaded.

• Driver class: the name of the JAVA class of the JDBC driver. • DB Uri: URI of the database used as cache.

• User name and password used to access the cache database. The user name should have read/write

privileges on the specified database, including permissions to create and delete tables and indexes.

• If Test connection is selected (recommended), the system will connect to the source to obtain its manufacturer and version number. These data will be used to optimize access to the source depending on its specific characteristics. If this option is not selected, a generic configuration will be used.

• Click on Pool configuration to configure the parameters of the connections pool used to access the cache

database (see section 5.3.1). The first time Virtual DataPort uses an external DBMS to store its cache data, it will generate the required tables and indexes automatically.



By default, views do not store its data in the cache. Therefore, after enabling the cache module, you also need to activate the cache option in the views that require it (see section 5.6.2) Note: When using an external DBMS to store the cache data, it must be started before the Virtual DataPort server.

Figure 98 Cache configuration

It is also possible to consult and/or configure the following cache function parameters:

• Maintenance Period: specifies (in seconds) the execution period of the cache maintenance task. The maintenance task is in charge of deleting the expired entries of the cache. If the value provided is 0 or negative, the maintenance task will never be executed.

• Time To Live: specifies (in seconds) the default expiry time of cache inputs.

If the cache is stored in the embedded database, some parameters can be configured by clicking on the “Embedded Derby Server Configuration” option:

• Enable Remote Access: it allows the embedded database to be accessed by external JDBC-compliant clients. This can be useful for auditing and monitoring purposes.

• Server port: this option allows changing the port where the embedded database runs. Likewise, to disable the cache simply select the “Cache off” option in the “Cache” panel of the “Server Configuration” tab. To invalidate partially or totally the data cached for a view, see sections 4.5.and 5.6.2. The configuration changes will become immediately effective except when the cache DBMS is changed, in which case the server must be restarted. . A different cache configuration can be specified for each DataPort server Database (see section 9.3.2).



8.1.4 Limiting the Number of Concurrent Requests

It is possible to limit the number of requests that the DataPort server will accept concurrently. When that limit is reached, the new requests will be queued and executed according to their arrival order. Limiting the number of concurrent requests is useful in high load environments, since it avoids performance degradation issues when there is a peak load. To configure this behavior, it is necessary to access the “Server Configuration” tab and choose the option “Concurrent Requests” (see Figure 99). It is then possible to configure the following parameters:

• Activate/deactivate the limitation of concurrent requests. If deactivated, the server will try to execute all requests as they come.

• Maximum number of concurrent requests. The Virtual DataPort server will only accept this number of

concurrent requests.

• Maximum number of requests on queue. Maximum number of queued requests waiting to be executed. New requests beyond this number will be discarded.

Figure 99 Concurrent Requests Configuration

8.1.5 Default Configuration of HTTP Proxy

Some VDP DataSources can use http routes to access source data (see section 5.5.1). It is possible to configure the default proxy preferences for these connections by accessing the “Server Configuration” tab and choosing the “HTTP Proxy” option (see Figure 100). The configurable parameters are:



• Host. Name of the machine that acts as Proxy. • Port. Port number in which the Proxy Server is being executed.

• Login. User identifier used to authenticate in the proxy. If the proxy does not require authentication, it

must be left blank.

• Password. User access password. If the Proxy does not require authentication, it must be left blank. In the http route used for each DataSource, it is possible to specify a Proxy configuration different from the default one. Please see section 5.5.1 for more information.

Figure 100 HTTP Proxy Configuration

8.1.6 Configuring the Wwapping to Disk Parameters

To avoid memory overflows, Virtual DataPort can swap to disk the intermediate results generated during the execution of a query. It can also swap the intermediate results of the sorting operations (views with ORDER BY) To configure the swapping parameters of a server, go to the “Server Configuration” tab and select the “Swapping” option (see Figure 101). To enable or disable swapping on a global level, use the options “Swapping status on” (default option) or “Swapping status off”, respectively.



Figure 101 Swapping Configuration

It is also possible to configure the following swapping parameters:

• Maximum size of each intermediate result (megabytes). During the execution of a query, Virtual DataPort will swap to disk an intermediate result produced during the execution, when that result exceeds the maximum size specified in this parameter. As a general rule, the specified value should not be greater than one third of the memory available for the JAVA virtual machine on which the DataPort server is run.

• Maximum size of the blocks written to disk (kbytes). Where DataPort swaps an intermediate result, it will write to disk using blocks of, at most, the size specified.

The general swapping configuration can also be specified for each Virtual DataPort database (see section 9.3.2). If the swap is globally enabled, it is possible to enable and configure it for one view (see section 5.6.3). It is also possible to configure it dynamically for a specific query using the VQL CONTEXT clause (see “Context clause” in the Advanced VQL Guide [VQL]). The configuration changes will become immediately effective.

8.1.7 Configuring Runtime Parameters for Stored Procedures

Virtual DataPort allows executing stored procedures written in JAVA (see section 5.7 and the ”Stored Procedures” section in Advanced VQL Guide [VQL]). Stored procedures can execute sentences on the DataPort server and process their results. The execution of those sentences is affected by several configuration parameters.



To configure the runtime parameters for stored procedures, go to the “Server Configuration” tab and choose the “Stored Procedures” option (see Figure 102).

Figure 102 Configuration of Stored Procedures runtime parameters

It is possible to configure the following parameters:

• Query Timeout: Maximum time (in milliseconds) the stored procedure will wait for the termination of a sentence. If the value 0 is specified, the procedure will wait indefinitely until the sentence ends.

• Chunk Size: The results obtained by executing a sentence can be divided into blocks, so it is not needed to

wait until a sentence ends to process the already obtained tuples. This parameter establishes the maximum number of results that a block can contain. If Virtual DataPort has obtained enough results to complete a block, they will be immediately returned to the stored procedure. The next results will be returned in new blocks. If this parameter is not specified (or receives the value 0), all the results of the query will be returned in a single block.

• Chunk Timeout: This parameter establishes the maximum time (in milliseconds) the server will wait

before returning a new block. If this time is surpassed, Virtual DataPort will return the current block even if it still does not contain the number of results specified by the Chunk Size parameter. If the Chunk Timeout parameter is not specified (or receives the value 0), then all the results of the query will be returned in a single block when the sentence execution ends.

8.1.8 Configuring Default Internationalization

The Virtual DataPort server internationalization configuration specifies aspects such as time zones, languages or currencies. For example, the default currency used in the Money-type values is specified by this property.



Virtual DataPort includes typical internationalization configurations and also allows new configurations to be created ad hoc (see “Creating New Internationalization Configurations” in the Advanced VQL Guide [VQL]). A specific internationalization configuration can also be specified for each DataPort server Database (see section 9.3.2) Note: it is also possible to specify a different internationalization configuration for each base relation of the system (see section 5.6.1). The Administration Tool also has its own internationalization configuration that can be changed in the ’View - Preferences - Locale‘ menu (see section 4.4.1)

8.1.9 JMS Listeners

Virtual DataPort can subscribe to a JMS server [JMS] to listen to VQL requests. Therefore, clients, instead of connecting to Virtual DataPort via JDBC, ODBC or a Web service, can send a request to a JMS server, which forwards it to Virtual DataPort. Then, the response is sent back to a queue or a topic of the JMS server, which forwards it to the client/s. E.g. a client sends a message such as ‘SELECT * FROM internet_inc WHERE iinc_id=1’ to the JMS server. The server will forward this to Virtual DataPort, which will send a response like: <?xml version="1.0" encoding="UTF-8"?> <response> <item> <iinc_id>1.00</iinc_id> <summary>Error in ADSL router</summary> <ttime>29-jun-2005 19h 19m 41s</ttime> <taxid>B78596011</taxid> <specific_field1>1</specific_field1> <specific_field2>1</specific_field2> </item> </response>

Figure 103 Response message sent by a JMS listener

If the request is a DML sentence such as ‘ALTER VIEW incidences CACHE INVALIDATE’, the response will be empty:

<?xml version="1.0" encoding="UTF-8"?> <response />

Figure 104 Response message to a DML query

Follow these steps to create a JMS listener:

1. Go to the ‘Server Configuration tab’ and click on ‘JMS Listeners’ (see Figure 105) 2. Click on ‘Add new listener’

3. Fill in the form:



a. Select ‘On’ to enable the listener. This option is useful because later, we can disable the listener without deleting it.

b. Select the database that the listener will connect to. JMS listeners, as JDBC clients, have to

connect to a specific Virtual DataPort database where they will execute queries.

c. Select the user name that Virtual DataPort will use to check if the listener has enough privileges to execute a query. E.g.: if we have two users:

• admin: is an ‘administrator’ so it can access any view of any database. • user1: is a ‘normal user’ that only has READ privileges over the database ‘samples’.

If we select the user ‘admin’, the listener can execute any query in the database selected in the box ‘VDP database’. However, if we select ‘user1’, we have to select the database ‘samples’ because that user can only access that database. Besides, the CREATE / UPDATE / DELETE queries will fail because ‘user1’ only has READ privileges.

4. Destination is the name of the queue or topic that Virtual DataPort will subscribe to, waiting for requests.

Depending on the vendor of the JMS server, we might have to create the JMS destination, or it will be created automatically when the listener tries to subscribe to it.

5. Select Queue or Topic.

6. Reply to is the name of the JMS queue or topic where the responses will be sent to. If empty, the

responses will be sent to the destination specified in the ‘Reply to’ field of the JMS request.

7. User name and Password are the credentials to connect to the JMS server.

8. JMS vendor. Select ‘GENERIC’ if the vendor of the JMS server is not in the list and it can be accessed via JNDI. In this case, we have to provide the appropriate JNDI connection properties (appendix 12.8 contains a list of the required JNDI properties for the most popular JMS vendors)

Note: the first time we create a JMS listener, we have to copy the client jars of the JMS vendor to the directory $DENODO_HOME/extensions/thirdparty/lib and restart the Virtual DataPort server. Section 12.8 contains a list of the client jars of the most popular JMS vendors.



Figure 105 Creating a new JMS listener

After creating the listener, we can see the list of existing JMS listeners (see Figure 106)



Figure 106 List of existing JMS listeners

By selecting the boxes beside the status of the listeners and clicking on ‘Enabled selected’ and ‘Disable selected’, we can enable / disable JMS listeners. To delete a listener, click on its name to open its configuration and then, click on the button ‘Delete’.

8.2 EXPORTING / IMPORTING THE SERVER METADATA

It is possible to export all server metadata to a VQL file, by using the ‘Export…’ option in the ‘File’ menu of the Administration Tool (see Figure 107). It exports, among other things:

• The metadata from every database of the server. • Users and their permissions.

• Configuration of JMS listeners.

This option is available only for “administrator” users.



Figure 107 Export dialog

After exporting the metadata of a Virtual DataPort server to a VQL file, it can be imported back to the same server or another one. This is especially useful for migration and backup purposes. The ‘Export’ dialog has the following options:

• Output file. Path of the new VQL file. • Elements to export. Select the elements that will be included in the VQL file:

o All Elements. Export all the elements of the server: data sources, views, Web services, users, JMS listeners, server settings, etc.

o Environment-specific elements. Only export the elements that depend on the server

environment: data sources, users, JMS listeners and server settings.

o Non-specific elements. Only export elements that do not depend on a certain environment.

In many organizations, installing a new application requires its validation in different environments (e.g. development, pre-production and production environments). Certain elements such as the paths or the authentication information used to access the data sources will normally be different in each environment. Usually, data sources are configured in each environment with the same name, but access the data source version available in each environment. Under these circumstances, it may be useful to export separately the elements that typically change from one environment to another and those that do not. For example, if users have created a new group of views in an environment and wish to pass it to another environment, they can export only the elements independent of the environment. The catalog elements considered



dependent on the environment are data sources (see section 5.3), server properties (such as ports), users, and databases (see section 9).

o Element. Select this option to export the metadata of a single element of the Virtual DataPort database that we are connected to. If the selected element type is ‘View’ we can select the ‘Export dependencies’ box to include in the output file, all the necessary elements to create this view (other views, types, data sources, etc)

• Replace / Drop elements. Select the action that a Virtual DataPort server will take when importing this

VQL file and the server has elements (data sources, views, etc) with the same name and type as the ones that is importing:

o Drop elements before creating them. Before creating a new element, it will delete, if exists, the element with that name and all the elements that depend on it. E.g. when a server imports a JDBC data source ‘ds_jdbc_acme’, first it will delete the existing JDBC data source ‘ds_jdbc_acme’ and all the views that depend on this data source. This option is useful when we want to make sure that the imported data sources do not have more views that the ones contained in the VQL file.

o Replace existing elements. It will replace the existing elements with the elements from the

file. E.g. when importing a LDAP data source ‘ds_ldap_activedirectory’ and the server already has a LDAP data source with that name, the server will replace this data source with the one from the VQL file. However, it will not modify or delete the views that depend on the existing data source.

o Do not replace existing elements. It will keep the existing elements and ignore the ones from the file. E.g. if the server has a view ‘internet_inc’ and the VQL file also contains a view with that name, the server will keep the existing view, instead of deleting it or replacing it. This option is useful when we need to add more elements to the server without affecting the existing ones.

• Include Jars. If selected, the output file will include the JAVA classes used to create stored procedures,

CUSTOM data sources and custom functions. See section 8.3 and the “Programming Extensions” section in the Advanced VQL Guide [VQL]).

• Include ITPilot scanners. If selected, the output file will include the scanners used by WWW (Web)

sources (see section 5.5.6 and ITPilot User Guide [ITPILOT]).

• Include ITPilot custom components. If selected, the output file will include the ITPilot custom components used by the WWW data sources.

• Enter Single Mode during Import. If selected, the file generated will begin with the statement ‘ENTER

SINGLE USER MODE‘ and finish with ‘EXIT SINGLE USER MODE‘, so that Virtual DataPort switches to single user mode while importing this file (see “Importing Metadata” section in the Advanced VQL Guide [VQL]). Note: you are strongly advised to always import metadata in a Virtual DataPort server in single user mode.

It is also possible to export the metadata from the active database only using the “Export Database” option in the ‘File’ menu. In this case, it is only required to have privileges over the database to export. The metadata from a specific database can be also exported from its configuration screen (see section 9.3.2).



Virtual DataPort also provides scripts to program the running of automatic backup copies (see section 11.3). There are three options to import the VQL file obtained in the export process:

1. Using the File – Import dialog.

2. Using the VQL Shell tool (see section 5.4.10)

3. Using the import / export scripts included in the utilities for importing and exporting metadata (see section 11.3)

8.3 IMPORTING EXTENSIONS

The implementation of stored procedures (see section 5.7), custom functions and Custom wrappers (see section 5.5.12) is done using JAVA code programming. See the “Programming Extensions” section in Advanced VQL Guide [VQL] for information and examples about how to create those elements. The File – Extensions – Jar Management option from the Administration Tool enables to add to the server new JAVA libraries (.jar files) that implement those elements.


Databases, Users and Access Rights in Virtual Dataport 156

9 DATABASES, USERS AND ACCESS RIGHTS IN VIRTUAL DATAPORT

This section describes various key concepts in the Virtual DataPort architecture. Section 9.1 describes the concept of databases in the context of a Virtual DataPort server. That is, logical groupings of datasources, views, base views, stored procedures and other elements in the server catalog. Section 9.2 describes the general concepts of user and access rights management in DataPort. Finally, section 9.3 describes how to manage this structure using the Administration Tool. The VQL commands for managing this structure are described in detail in the VQL Advanced Guide [VQL].

9.1 DATABASES IN VIRTUAL DATAPORT

A Virtual DataPort server can contain several databases (do not confuse with the possible external databases that can act as data sources of the system). A Virtual DataPort database represents a virtual schema comprised of a series of DataSources, views and stored procedures. Each database is independent of the other databases of the server and, as described in detail in the next section, different users can have different privileges for each database. When a Virtual DataPort server is installed, an initial database called admin is created.

9.2 USER AND ACCESS RIGHTS STRUCTURE IN VIRTUAL DATAPORT

9.2.1 User Types

Denodo Virtual DataPort distinguishes two types of users:

• Administrators. These can create, modify and delete databases in a DataPort server without any limitation. Likewise, they can also create, modify and delete users. When the server is installed, a default administrator user is created whose name is admin and whose password is also admin. This user can never be deleted.

• Normal users. These cannot create, modify or delete users. They cannot create or delete databases,

although they can have connection, read, create or write privileges to one or several databases or to specific views contained therein.

9.2.2 Types of Access Rights

Virtual DataPort access rights are applied to a specific user to delimit what tasks he/she is allowed to perform on databases, views and stored procedures. Access rights for a user can be applied globally to a database or specifically to a view or stored procedure in a specific database. Access rights to particular views and stored procedures are applied only if the user does not have the corresponding access right on a global level. Denodo Virtual DataPort supports the following types of global database access rights:

• Read access: Users with read access to a database on a global level can:



o View the list of base relations, views and/or stored procedures of the database (corresponds to the VQL LIST command). If a user does not have read access to a database, but does have it for some of its views and/or stored procedures, the LIST command may be executed, but it will only display the group of views and procedures to which the user has read access.

o View the schema of any base relation, view or stored procedure of the database (corresponds to the VQL DESC command).

o Execute queries on any view and/or stored procedure of the database (corresponds to the VQL SELECT command).

• Creation access: Users with creation access to a database can:

o Create DataSources, views, base relations and stored procedures in the database (corresponds to the VQL CREATE command).

• Write access: Users with write access to a database can execute the following tasks: (write access

automatically implies read access too): o Delete any view, base relation and/or stored procedure of the database.

Users can also delete any DataSource created by them, but not DataSources created by other users (corresponds to the VQL DROP command).

o Modify any view, base relation and/or stored procedure from the database. Users can also alter any DataSource from their own databases, but not alter DataSources created by other users (corresponds to the VQL ALTER command)

• Connection access: If a user has this access to a database, then he/she can connect to it. This access right

is useful, for example, if you wish to temporarily revoke a user’s access to a database without having to alter the rest of his/her normal privileges.

Denodo Virtual DataPort also supports individual privileges to specific views and stored procedures. The privileges that can be applied to a specific view and/or stored procedure of a database are:

• Read access: Users with this privilege on a view or stored procedure can: o View its schema (corresponds to the VQL DESC command). o Execute queries to it (corresponds to the VQL SELECT command). o Create new views that use it (only if the user has creation rights in the database). Corresponds to

the VQL CREATE VIEW command. o If a user does not have read access to a database, but does have read access to some of its

views and/or procedures, the LIST command may be executed, but only those components will be displayed.

• Write access: Having write access implies read access too. If a user has this privilege on a view or stored

procedure, he/she may execute the following additional tasks on it: o Delete the component (corresponds to the VQL DROP command). o Edit the component (corresponds to the VQL ALTER command).

• Insertion access: Having this privilege on a view allows inserting tuples in it (this corresponds with the

INSERT VQL statement). It is not applicable to stored procedures. • Update access: Having this privilege on a view allows updating tuples from it (this corresponds with the

UPDATE VQL statement). It is not applicable to stored procedures. • Deletion access: Having this privilege on a view allows deleting tuples from it (this corresponds with the

DELETE VQL statement). It is not applicable to stored procedures.



9.3 GRAPHIC ADMINISTRATION OF DATABASES, USERS AND ACCESS RIGHTS

To manage the databases, users and access rights of a Virtual DataPort server, access the administration tool in server administration mode, which implies doing it with an administrator-type user (in this case, it is not mandatory to specify a database in the server connection uri). When the server is installed, a default administrator user is created whose user name is admin and whose password is also admin. The following sections describe how to create/modify/delete databases and how to create/modify/delete users.

9.3.1 Creating Databases

To create a new database, open the “Database Management” tab and press the ‘new’ button.

Figure 108 Creating a new database

The tool will request the following data for the new database (see Figure 108):

• Name of the database. • Description. Optional field to include a description of the database.

9.3.2 Configuration and Deletion of Databases

To view the current list of databases of the Virtual DataPort server open the “Database Management – Databases” tab. For each database, its name and description are displayed (see Figure 109). The value of the description field may be modified by pressing the “edit” button.



Figure 109 Description of a database

The available actions for a database are:

• Click the “Assign privileges” button to obtain a list of all Virtual DataPort users who are not administrators. For each user, you can specify whether the user has global connection, creation, read and/or write access rights to the database (see Figure 110). For more information about user privileges, read section 9.3.4.2.

• The “i18n”, “Swap” and “Cache” buttons open the dialogs to change the internationalization settings (see

section 8.1.8), swapping policies (see section 8.1.6) and cache settings (see section 8.1.3), respectively, for the database. In all three cases, if the “Default Configuration” box is selected, the Server Configuration will be used.

• By pressing the “Export VQL” button, all database metadata can be exported to a VQL file so that it can be

easily recreated in another Virtual DataPort installation. Neither the users nor its access rights for the database are included in the export process. The database creation statement is not included either. We can also export all the metadata of the server, instead of just one database, using the ’File – Export’ option (see section 8.2).

• Delete the database (button “Delete”). Due to the delicacy of the operation, the Tool requests the user’s

password again) Important: when a database is deleted, all its components are eliminated: DataSources, views, base relations, etc.



Figure 110 Assigning privileges on a database

9.3.3 Creating Users

To create a user, open the “User Management” tab and press the ‘New User’ button. The tool then requests the following data for the new user (see Figure 111):

• User login. • User password. It should be written twice so that it can be checked. • Description. Optional field to include a description of the user. • Type of user: Users can be of the type Administrator or Normal User (user without administration

privileges). • Authentication type. Users can be authenticated against DataPort (“Normal” option) or against an LDAP

server registered in DataPort (see section 5.5.10 for information on how to register an LDAP server). The data to be completed is different, depending on the option chosen.

o In the “Normal” case, a password will be requested for the user. It should be written twice so that it can be checked.

o In the case of “LDAP”, the following data must be provided (see Figure 111): LDAP server to use. Select the database where the LDAP server required has been

registered using the selectable “Database”. Once this has been done, select the LDAP data source using the selectable “Datasource”.

LDAP user. The name of the user in the LDAP server. For example, the value 'cn=test,ou=People,dc=denodo,dc=com’ identifies the test user in an organizational unit People for the domain denodo.com.



Figure 111 Creating a normal-type user

NOTE: If a LDAP datasource is deleted on cascade (see section 5.3.2), then the users depending on it will also be deleted. This operation can only be executed by an administrator user.



Figure 112 Creating an LDAP-type user

9.3.4 Modifying and Deleting Users

To see the current list of users of the Virtual DataPort server go to the “User Management” tab of the administration tool. For each user the login, description field and type are shown (administrator or normal user). See Figure 113.



Figure 113 Description of a user

The user description and password can be changed by pressing the “edit” button. If the user is not an Administrator, the option “Assign privileges” may be used to modify his/her privileges for the various DataPort databases (see section 9.3.4.2). From the list of users screen, you can also delete the user by pressing the “delete” button (the system will request confirmation of the operation). The predefined “admin” administrator cannot be deleted.

9.3.4.1 Changing Active User Password

The active user password can be modified using the ‘Change password’ option on the application ‘File’ menu. This option is available both in database administration mode and in server administration mode.

9.3.4.2 Modifying the Privileges of a User

The privileges of the user for the different databases and views can be modified by using this option (this option is only available for users that are not administrators). Once the user’s privileges edit screen has been accessed (see Figure 114), a list of the VDP server databases appears. For each database, a series of checkboxes allows assigning the user the desired global access rights on each database.



Figure 114 Assigning privileges to a user at database level

The privileges that can be assigned to a user in a database are:

• Connect. The user can connect to the database. Note: It is very important to grant this privilege to a user. Otherwise, the other privileges will be ignored. This privilege is useful to deny all the privileges of a user temporarily. • Create. The user can create new elements in the database: data sources, views, publish web

services, etc.

• Read. The user can access all the views and stored procedures of the database.

• Write. The user can modify / delete any view / stored procedures of the database. Write access implies read access.

• Admin. The user has the permission to connect, create, read and write to the database. Besides

the user can: o Set the configuration parameters of the database: I18N, cache, swap, etc. o Edit the description of the database. o Grant / revoke privileges to normal users (not admin or database-admin users). It cannot

grant the Admin privilege to other users.

A user with this privilege cannot: o Create / delete users. o Change users’ password o Create / drop databases o Grant Admin privileges to other users.



To configure the privileges on specific views and stored procedures of a database, click the “edit” link of the “Advanced privileges” column. This dialog contains a list of the views and stored procedures of the database (see Figure 115). For every view or stored procedure, there is a check box allowing to provide users with read, write, insert, update and/or delete access. Note: if you have assigned the user a certain privilege on a global level for the whole database, then the rights assigned to specific views and stored procedures will be ignored.

Figure 115 Assigning privileges to a user at view and/or stored procedure level


Monitoring the Virtual DataPort Server 166

10 MONITORING THE VIRTUAL DATAPORT SERVER

The Denodo Platform provides two different ways of monitoring the servers of the Platform: Virtual DataPort, Scheduler and ITPilot’s browser pool and maintenance:

1. Using a Java Management Extensions (JMX) Agent such as jConsole. See section 10.1. 2. Using the Denodo Monitor (application included in the Denodo Platform). See section 10.2.

10.1 MONITORING WITH A JAVA MANAGEMENT EXTENSIONS (JMX) AGENT

It is possible to access monitoring information of the server using the Java Management eXtensions (JMX) standard [JMX]. This information can be used to control the use of the server and audit the actions carried out on the data sources and/or the Virtual DataPort metadata. The available monitoring information is:

• General information on the server: number of active and inactive connections, memory usage, etc. • Configuration and accesses to the server cache. • Accesses to each data source. • Accesses to the views and stored procedures. • Transactions executed by the DataPort server.

DataPort also generates different types of JMX notifications whenever:

• A DDL statatement is executed. E.g. ALTER TABLE, CREATE VIEW, etc. • A DML statement is executed. E.g. SELECT, INSERT, etc. • Transactions start or end. • …

In JMX, the information exported is displayed as a set of objects known as MBeans. The MBeans exported by DataPort belong to the JMX domain com.denodo.vdb.management.mbeans. Any JMX client can be used to access the DataPort monitoring information and events. JMX jConsole (included with the Java Developer Kit 1.5 and above) and jManage [JMANAGE] consoles, among others, have been successfully tested to access the information and events exported by the DataPort server. To use these consoles, it is usually needed to specify an URL to the server. For instance, if the DataPort server is running in the port 9999 in the local host, the URL would be: service:jmx:rmi:///jndi/rmi://localhost:9999/jmxrmi It is also possible to access the MBeans from an external application, using the JMX API [JMX]. Section 10.1.1 provides information on how to use jConsole to access DataPort. The following subsections describe in further detail the information and events available in each of the following categories: general server information, general data source information, general cache information, information and events on DDL statements, information and events on DML statements and information and events on transactions.



10.1.1 Using JConsole

jConsole is a server administration console compliant with the JMX standard that is distributed with the JDK 5 or higher. This section introduces the use of jConsole for accessing DataPort monitoring information and events. Refer to the JDK documentation for detailed information on the use of jConsole [JCONSOLE]. To run jConsole, execute: %JAVA_HOME%\bin\jconsole Note: jConsole is not included with the JRE, only with the JDK. In the agent connection window, select the ‘Remote’ tab, enter the DataPort server data (machine, port, user name and password) and click on the ‘Connect’ button. An administrator-type user must be used.

Figure 116 Connection to DataPort from jConsole

On connecting to the server, jConsole displays the default information for a monitored agent such as the server’s uptime, number of threads or memory usage.



The specific DataPort server information is in the ‘MBeans’ tab. On the left side of the screen, there’s a list of the available MBeans. The MBeans contained in com.denodo.vdb.management.mbeans provide information about the Virtual DataPort server. VDBServerManagementInfo (see Figure 117) show, among other things, the number of active connections and the number of total connections since the server was started. The values of the properties that can be modified are displayed by jConsole in blue (e.g. the value of the CompleteManagementActive property is blue).

Figure 117 Virtual DataPort MBeans

The information and events provided by each MBean is described in the following sections. This section only shows an example to illustrate the use of jConsole with DataPort. Example: access to the information and events regarding active connections and DDL statements. On the left of Figure 117 are the MBeans containing the information on DataPort. All of these are uploaded to the domain com.denodo.vdb.management.mbeans. On the right of the figure are the properties of the MBean com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo (see section Figure 120), which include the number of active connections and the number of total connections, since the server was started. The values of the properties that can be modified are displayed by jConsole in blue (e.g. the value of the CompleteManagementActive property is blue).



The information and events regarding DML statements (SELECT, INSERT, UPDATE, DELETE, etc.) are accessible via the MBeans uploaded to com.denodo.vdb.management.mbeans:type=RequestsManagementInfo (see section 10.1.6). By clicking on this name, a list of the existing databases in the DataPort server will be displayed. By clicking on one of them, information on the DML statements run on this database will be accessed. For example, Figure 118 shows information on the queries run on the admin database. Information such as the total number of requests executed (TotalRequests property) or the number of active requests at that time (ActiveRequests property) is displayed. Click on the ‘Refresh’ button to refresh the information on the attributes displayed on the current screen. Note, however, that each DDL request on DataPort appears as a new attribute. For jConsole to display new attributes (and, therefore, data corresponding to new requests), you must access the screen of any other MBean and then return to the original screen. This will ensure jConsole loads the new attributes. To view the data on each request, double click on its ‘Value’ cell. The data about each statement includes the type of statement executed (SELECT, INSERT, etc.), the user who has run it, the time at which it was run and the exact statement run (see Figure 119).

Figure 118 Attributes tab of the RequestsManagementInfo Mbean



Figure 119 Details of a query in jConsole

Continuing with the example, the events related to DML statements from the admin database can be seen by clicking on the ‘Notifications’ tab and then the ‘Subscribe’ button. From then on, every time a DML statement is run in DataPort, a new notification will be displayed in jConsole. Double click on its ‘UserData’ cell for details on the statement run (see Figure 120).



Figure 120 Details of a notification in jConsole

10.1.2 General Information on the Server

General information about the server is obtained through the Mbeans: • com.denodo.vdb.management.mbeans.VDBServerManagementInfo • com.denodo.vdb.management.mbeans.MemoryManagementInfo • com.denodo.mbeans.LogManagementInfo

The properties of each one are described below:

• com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo. o CompleteManagementActive: If true, complete monitoring mode is enabled. In that

mode, an estimate of the current memory (TotalMemory property of the MBean MemoryManagementInfo) and the memory peak occupied by a query (MaxMemory property of the MBean MemoryManagementInfo) is calculated. The monitoring of these parameters is costly and, therefore, this mode should only be enabled when necessary. This property can be modified by the user and can take the values true or false.

o ServerName: Virtual DataPort server URI (e.g. //localhost:9999/VQLServer) o TotalConnections: Total number of connections established with the server, since it was

started.



o ActiveConnections. Number of currently active connections.

• com.denodo.vdb.management.mbeans:type=MemoryManagementInfo. The attributes of this MBean are only calculated if complete monitoring mode is enabled (see above).

o TotalMemory. Estimate of the total memory occupied by the server. o MaxMemory. Estimate of the memory peak reached by the server since its start. o gc () . Operation that executes the “garbage collector” of the JAVA virtual machine of the Virtual

DataPort server.

• com.denodo.mbean.LogManagementInfo. It has operations to obtain and change the appenders of Log4J [LOG4J]

o getLogLevel(<logger name>).Obtains the log level of a logger. I.e: “com.denodo” o setLogLevel(<logger name>, { DEBUG | INFO | WARN | ERROR })

Sets the log level of a logger.

10.1.3 General Information on Data Sources

General information about data sources is obtained through the MBean: com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo,name=GlobalDataSourceManagementInfo This MBean has the following attributes:

• TotalDataSourceCount. Number of datasources • TotalActiveRequestCount. Number of current requests to a datasource. • For every type of datasource there is an attribute with the number of datasources and the number of active

request to that type of source. • ITPMaintenanceDataSourceCount. Number of Web (WWW) sources that have its

maintenance option enabled. • ITPMaintenanceActiveRequestsCount. Number of current requests to Web sources that

have its maintenance option enabled. Each VDP database has its own MBean with the same attributes, but only considers data sources that belong to that database: com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo,databasename=<VDP database name>,aux=LocalDataSourceManagementInfo Besides, each datasource has its own MBean: com.denodo.vdb.management.mbeans:type=DataSourceManagementInfo, dataSourceType=<dstype>,dataSourceName=<dsname>. where <dstype> is the type of data source (JDBC, ODBC, WS, XML, etc.) and <dsname> is the name of the data source. These MBeans have the following attributes:

• DatabaseName. Name of the data source database. • DataSourceName. Name of the data source. • DataSourceType. Type of the data source: JDBC, ODBC, XML, WS (Web services), DF (delimited

files), ARN (Aracne), GS (Google Mini), CUSTOM, LDAP, ITP (Web). • NumRequests. Number of requests made on the data source since the start of the server. • ActiveRequests. Number of active requests to the data source.



• MaxActive, NumActive and NumIdle (only for JDBC and ODBC sources). Provide information about the pool of connections with the database.

o MaxActive: maximum number of connections in the pool. o NumActive: number of connections established with the source that are being used to

execute a query. o NumIdle: number of idle connections established with the source.

• isMaintenanceEnabled (only for Web sources). True if the maintenance option is enabled.

10.1.4 Information on the Cache

General information on the server cache is obtained through the MBeans:

• com.denodo.vdb.management.mbeans:type=Cache,databaseName=<dbname>,subtype=DatabaseCache, where <dbname> specifies the name of a DataPort database, and

• com.denodo.vdb.management.mbeans:type=Cache,databaseName=<dbname

>,subtype=ViewCache,viewName=<vname>, where <dbname> specifies the name of a DataPort database and <vname> specifies the name of a view of said database.

Below is a description of the properties of each one of these two subtypes:

• subtype=DatabaseCache. The following properties are given for each DataPort database: o DatabaseName. Name of the database. o CacheStatus. Status of the cache: ON or OFF o UserName. Name of the user in the data source associated with the database used as cache. o UserPwd. Password of the user in the data source associated with the database cache. o DriverClassName. Class of the JDBC driver in the data source associated with the

database used as cache. o LastCacheAccessStatus: Indicates whether the last access to the cache database was

successful (true) or not (false). o CustomDataSource: (read only) This indicates whether the database used as cache uses

(true) or not (false) a different data source to that used by default by the DataPort server. o DBURI: Connection URI to the database used as cache. o isCustomDataSource(). Operation that returns true if the database uses a different

cache configuration that the DataPort server. False otherwise.

• subtype=ViewCache. The following properties are given for each view (viewName) of every database (databaseName):

o DatabaseName. Name of the DataPort database to which the view belongs. o ViewName. Name of the view. o CacheStatus. Status of the view cache (ON/OFF/POST). o TTLInCache. Lifetime of the data in the view cache. o LastAccess. Moment of the last access to the view cache. o LastRefresh. Moment in which new tuples were last written in the view cache. o RefreshFailuresCount. Number of failures when writing new tuples in the view cache.

10.1.5 Information and Events on Catalog Access (DDL Statements)

The information and events about the DDL statements is obtained through the MBean:



•

com.denodo.vdb.management.mbeans:type=CatalogManagementInfo, databaseName=<dbname> where <dbname> specifies the name of a DataPort database. The MBeans of this type offer a single property (read only) called DatabaseName that indicates the name of the database. On subscribing to the MBean of a database, a notification is received every time one of the following events occurs:

• A statement affecting the metadata of the database is executed (e.g. a new view is created, a view definition is modified, the privileges of a user are changed, etc).

• A transaction starts. • A transaction ends.

The following fields appear in each notification:

• Timestamp. Moment in which the notification of the statement is generated in the JMX server. • Type. Type of the transaction: start of a transaction (‘startTransaction’), end of a transaction

(‘endTransaction’) or execution of a DDL statement (‘newEvent’). • UserData. Compound element. Its subproperties depend on the type of notification:

o Statement executed: DatabaseName: Name of the DataPort database on which the statement is run. EventID: ID of the notification generated by DataPort. Time: Moment at which the statement is run according to the DataPort server. TransactionID: ID of the transaction within which the statement is run. Type: Name of the executed operation. UserName: ID of the user running the statement. VQLQuery: VQL code for the statement.

o Transaction start: Autostarted: Indicates whether the transaction has been explicitly created by the

user. All statements modifying the DataPort catalog must be run within a transaction. Therefore, if the user has not created it, DataPort will create it itself.

DatabaseName: Name of the DataPort database on which the transaction is started.

EventID: ID of the notification generated by DataPort. The IDs are allocated using a counter.

Time: Moment at which the transaction starts according to the DataPort server. TransactionID: ID assigned to the starting transaction. UserName: ID of the user starting the transaction.

o Transaction end: DatabaseName: Name of the DataPort database on which the transaction is run. EventID: ID of the notification generated by DataPort. The IDs are allocated using a

counter. Status: Completion status of the transaction (COMMIT or ROLLBACK). Time: Moment at which the transaction finishes according to the DataPort server. TransactionID: ID assigned to the ending transaction. UserName: ID of the user running the transaction.

• SeqNum: Sequence number generated by DataPort for the notification. Normally, the sequence numbers

are assigned using a counter. Message:



o If the execution of a statement is notified: Processed the <event eventId> in transaction <transactionId>.

o If the start of a transaction is notified: Started the transaction <transactionId>.

o If the end of a transaction is notified: Finished the transaction <transactionId>.

• Event: This will be javax.management.Notification[com.denodo.vdb.management.mbeans:type=CatalogManagementInfo,databaseName=<dbName>][type=<eventType>][message=<eventmessage>], where <dbName> is the name of the DataPort database, <eventType> is startTransaction, endTransaction or newEvent and <eventmessage> takes the same value as in the Message property.

• Source: MBean name.

10.1.6 Information and Events on the Running of DML Statements

The information and events about the executed DML statements can be obtained through the MBean: com.denodo.vdb.management.mbeans:type= RequestsManagementInfo,databaseName=<dbname>, where <dbname> specifies the name of a DataPort database. The following properties are given for each database:

• DatabaseName. Name of the DataPort database. • MaxRequests. Maximum number of requests exported at the same time in the MBean. Each request

appears as an attribute of the form Request. • TotalRequests. Total number of requests recorded since the start of the server. • ActiveRequests. Number of active requests. • Request. Compound element. Its subproperties are:

o Cache. True if the query has accessed the cache during its execution. False otherwise. o Completed. True if the query finished correctly. False otherwise. o DatabaseName. Name of the DataPort database on which the statement is executed. o EndTime. Moment at which the statement finished its execution. o Identifier. Transaction identifier. o MaxMemory. Estimate of the memory peak consumed by the query. Only available if the

CompleteManagementActive of the MBean com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo is enabled.

o Memory. Estimate of the memory consumed by the statement. Only available if the attribute CompleteManagementActive of the MBean com.denodo.vdb.management.mbeans:type=VDBServerManagementInfo is enabled.

o NumRows. Number of rows returned by the query. o RequestType. This indicates the type of statement. It can take the following values:

SELECT BASE VIEW, SELECT VIEW, QUERY WRAPPER, CALL STOREDPROCEDURE, INSERT and UPDATE.

o SessionId. Id of the session that this request belongs to. o StartTime. Moment at which the statement started its execution. o State. It can take the following values: OK, ERROR or PROCESSING. o Swap. True if the server swapped the intermediate results to disk during the execution of the

query. False otherwise.



o UserName. ID of the user running the statement. o VQLQuery. VQL code for the statement. o WaitingTime. Time that request was waiting in the queue before being processed.

Furthermore, the MBeans of this type also allows subscribing to notifications. Every time a DML statement is run on the specified database, two notifications are received: one, indicating the beginning of the request and the other indicating the end.

• Timestamp. Moment in which the notification is generated in the JMX server. • Type. This type of notification includes the string ‘startRequest’ or ‘endRequest’ as the type

indicator. • UserData. Compound element. Its subproperties are the same as those of the Request

property described above. • SeqNum. Sequence number generated by DataPort for the notification. • Message. In this type of notification, it takes the value ‘Started the request’ or ‘Finished

the request’. • Event. This will be

javax.management.Notification[source=com.denodo.vdb.management.mbeans:type=RequestsManagementInfo,databaseName=<dbname>][type=endRequest][message=Finished the request ''], where <dbName> is the name of the DataPort database and is the statement identifier.

• Source. MBean name.

10.1.7 Information and Events on Transactions

The information and events about the transactions can be obtained through the MBean: com.denodo.vdb.management.mbeans:type= TransactionsManagementInfo,databaseName=<dbname>, where <dbname> specifies the name of a DataPort database. The following properties are given for each database:

• DatabaseName. Name of the DataPort database. • MaxTransactions. Maximum number of transactions exported at the same time in the MBean. Each

transaction appears as an attribute in the form Transaction. • TotalTransactions. Total number of executed transactions since the start of the server. • ActiveTransactions. Number of currently active requests. • Transaction . Compound element. Its subproperties are:

o Autostarted. Indicates whether the transaction has been explicitly created by the user. All statements modifying the DataPort catalog must be run within a transaction. Therefore, if the user has not created it, DataPort will create it itself and, in this case, the value of this property will be false.

o DatabaseName. Name of the DataPort database on which the transaction is run. o EndTime. Moment at which the transaction ended. o Identifier. Transaction ID. o StartTime. Moment at which the transaction started. o State. This indicates the status with which the transaction ended. It can take the following

values: ROLLBACK or COMMIT. o UserName. ID of the user running the transaction.



It is possible to subscribe to the events of the TransactionsManagementInfo MBean. In that case, every time a transaction starts or ends on the specified database, a notification with the following data is received:

• Timestamp. Moment in which the notification is generated in the JMX server. • Type. This type of notification takes the string ‘startTransaction’ or ‘endTransaction’ as

the type indicator, depending on whether the notification indicates the start or the end of a transaction. • UserData. Compound element. Its subproperties are the same as those of the Transaction

property described above. • SeqNum. Sequence number generated by DataPort for the notification. Normally, the sequence numbers

are assigned using a counter. • Message. If the notification indicates the start of a transaction, its value is ‘Started the

transaction’. If it indicates the end of a transaction, its value is ‘Finished the transaction’.

• Event. This will be javax.management.Notification[source=com.denodo.vdb.management.mbeans:type=TransactionsManagementInfo,databaseName=<dbname>][type=<type>][message=<msg>], where <dbName> is the name of the DataPort database, <type> is the type of notification (‘startTransaction’ or ‘endTransaction’) and <msg> is the notification message (‘Started the transaction’ or ‘Started the transaction’).

• Source. MBean name.

10.2 DENODO MONITOR

The Denodo Monitor is a tool included in the Denodo Platform that logs several parameters of the Denodo servers: Virtual DataPort, Aracne and Aracne Index [ARCN], ITPilot’s browser pool and maintenance [ITPILOT] and Scheduler [SCHED]. Denodo Monitor includes several types of “monitors” which obtain information from each server. There is a monitor to log the active processes and connections of the local computer, memory resources used by each Java Virtual Machine, etc. The Denodo Monitor is located in the directory $DENODO_HOME/tools/monitor, compressed in a zip file. To execute it, decompress this file and execute one of the following commands:

a) denodomonitor_startup.sh/bat Launches de Denodo Monitor and keeps it running indefinitely. Periodically, it obtains information from all the “monitors” and logs it into text files located in the denodo-monitor\logs directory.

b) denodomonitor_startup.sh/bat -s

Makes a single execution of all the “monitors”, prints the results in the logs and exits. Note: in this mode, the “VDP queries monitor” is not launched, even if it is configured to do so.

c) denodomonitor_startup.sh/bat -h { vdp | browserpool | arn | arnindex | maintenance | scheduler } It obtains a heap dump of one of the Denodo servers by connecting to the JVM of that server via JMX and requests it to make a memory dump. The memory dump will be stored in the server machine’s temporal directory.

Note: in Unix environments, give execute permissions to the script: chmod +x bin/denodomonitor_startup.sh There are three different types of “monitors”:



• Local monitors. They gather information about the machine where the Denodo Monitor is running. • Server monitors. They obtain monitoring information by connecting to a server via JMX [JMX]. Thus, they

can obtain this information from remote servers.

• VDP Queries monitor. It logs several parameters of all the VQL queries received by a VDP server. It also obtains this information via JMX, so it can also monitor remote servers.

10.2.1 Local Monitors

Local monitors execute commands in the local computer to obtain certain information.

10.2.1.1 Processes Monitor

This monitor logs information about all the running processes in the local computer. In Windows OS, it logs the output of the command ”tasklist.exe” and in Linux, the output of “ps”. The log file is stored in conf/processes.log.

10.2.1.2 Sockets Monitor

This monitor logs the information of the active connections of the local computer by executing the command "netstat". The log file is stored in conf/sockets.log.

10.2.2 Server Monitors

These monitors connect to a Denodo server via JMX. The difference with the local monitors is that local monitors can only obtain data of the machine they are executed in, while server monitors can obtain information remotely.

10.2.2.1 Resources Monitor

This monitor logs several parameters of the memory footprint of the server. The log file is stored in conf/<server name>-resources.log Every time the information about resources is logged, the following information is collected:

• Memory information: various parameters of the memory usage of the JVM that executes the monitored server:

o Perm Gen o Survivor Space o Tenured Gen o Eden Space o Code Cache o Heap Memory Usage o Non Heap Memory Usage

• Classes information: o Loaded Class Count. Number of currently loaded classes. o Total Loaded Class Count Total. Total number of loaded classes. o Thread Count. Number of threads. o Peak Thread Count. Maximum number of threads.

• VDP Information (only for monitors connected to a Virtual DataPort server)



o VDP Total Conn. Number of connections opened during the interval. o VDP Active Conn. Number of currently opened connections. o VDP Active Requests. Number of request that are currently being executed. o VDP Waiting Requests. Number of enqueued requests. o VDP Total Mem. Estimate of the total memory occupied by the server. o VDP Max Mem. Estimate of the memory peak reached by the server since its start.

10.2.2.2 Threads Monitor

This monitor logs basic information and CPU usage of every thread of the server. It is very useful to identify VQL queries that consume too much CPU time. The log file is stored in conf/<application name>-threads.log.

10.2.3 Virtual DataPort Queries Monitor

The Virtual DataPort Queries monitor is specific to Virtual DataPort. It subscribes to the RequestsManagementInfo MBean (see section 10.1.6), which sends a notification every time Virtual DataPort receives a query. The monitor stores the information contained in these notifications, in $DENODO_HOME/tools/monitor/conf/vdp-queries.log.

10.2.4 Configuring the Denodo Monitor

The configuration file of the Denodo Monitor is $DENODO_HOME/tools/monitor/conf/ConfigurationParameters.properties. Almost all the properties in this configuration file are self-explanatory, so we are going to review only the most important ones:

• monitors.local. Comma-separated list of local active “monitors”: processes (10.2.1.1) and sockets (10.2.1.2).

• monitors.remote. Comma-separated list of servers to monitor: vdp, browserpool, arn,

arnindex, maintenance or scheduler.

• {arn | arnindex | browserpool | maintenance | scheduler | vdp}.monitors. Properties that indicate which remote “monitors” will be active for each server. The possible values for these properties are: vdpqueries (only for the property vdp), threads and resources.

• {processes | sockets | resources | threads}.interval. Number of seconds between

two JMX requests issued by that monitor.

• vdpqueries.snmptrapagent.enable. If true, the monitor sends SNMP traps with the information received by the Queries monitor. To use it, uncomment and configure the properties vdpqueries.snmptrapagent.

• The properties .jmx.password store the password to connect to the server of that property. These

passwords can be stored encrypted. To do this, set the value of the property .jmx.password.encrypted to true if the value of .jmx.password is encrypted. For example, by default the Virtual DataPort password is admin. If you changed it and you want to store it in this configuration file encrypted, do the following:

1. Execute the script $DENODO_HOME\tools\monitor\denodo-monitor\bin\encrypt_password <new password>



The output of this script is the encrypted password that you have to put in the configuration file of the Denodo Monitor.

2. Open the file $DENODO_HOME\tools\monitor\denodo-

monitor\conf\ConfigurationParameters.properties

3. Set the value of the property vdp.jmx.password to the encrypted password you obtained in step 1.

4. Set the value of the property vdp.jmx.password.encrypted to true.


Cluster Architectures / Server Backup 181

11 CLUSTER ARCHITECTURES / SERVER BACKUP

Virtual DataPort can operate within a cluster architecture to provide high availability and load balancing. Typically, clustering architectures for DataPort operate as follows:

1. An external load-balancing system distributes the requests received among a pool of Virtual DataPort servers.

2. The load-balancing system detects when a server is not responding (using the so-called ‘server health

check’). In this case, the load is distributed among the remaining servers.

3. The clustering system detects when the server is available again (using the ‘health check’) and adds it to the pool.

The following clustering solutions have been tested with Virtual DataPort:

• Linux Virtual Server architecture [LVS], using Keepalive [KALV] to check whether a server is responding at any given time.

• Windows 2000 Network Load Balancing Tool [WNLB]. This is the clustering solution included in the

Windows 2000 server family. Virtual DataPort provides some features to make cluster architecture configuration easier:

• It provides a ‘PING’ script to check the status of a DataPort server. Although solutions such as Keepalived provide their own default implementations for ”health-checking” an application, the PING script provided by DataPort detects failure situations that would not be detected by other solutions.

• It provides a utility to automatically replicate the metadata and the configuration of a DataPort server in a

pool of other DataPort servers. This utility can also be used to schedule backup copies of the metadata of a DataPort server.

The following sections describe how to:

1. Install a set of tools that help building a cluster with several Virtual DataPort servers (section 11.1). 2. Use the PING script (section 11.2)

3. Replicate the metadata of a Virtual DataPort server, to a group of servers (section 11.3).

11.1 TOOLS INSTALLATION

The tools are located in the path %DENODO_HOME%\tools\db\denodo-db-tools.zip. To use them, decompress this file.

11.2 USE OF THE PING SCRIPT

The script ping is located in the directory %DENODO_HOME%\tools\db\denodo-db-tools\bin. The syntax of this script is:



ping [-t timeout] [-v] [//]host[:port], where host and port are the machine and port where the Virtual DataPort server runs (port 9999 is assumed by default, if it is not specified) and timeout is the maximum time (in milliseconds) to wait for the server response. Option –v shows via the standard output the status of the server and the time taken to receive a response from it. The ping script returns 0 if the status check is successful. Otherwise, it returns another value. An example of running the ping command is shown below: ping -t 5000 -v //localhost:9999/ This command checks the status of a Virtual DataPort server running in the local machine, listening on port 9999. The script waits at most 5 seconds for the server’s response. The result of the check is displayed by the standard output.

11.3 USE OF THE IMPORT/EXPORT SCRIPTS FOR BACKUP AND/OR REPLICATION

The import and export scripts are part of the “Denodo Tools” toolkit ($DENODO_HOME/tools/db/denodo-db-tools.zip) The export script exports the metadata of a Virtual DataPort server to a file. We can export all the metadata of the server and its configuration or just the metadata of a specified database. The output of the script is equivalent to the ‘Export’ and ‘Export database’ output of the Administration Tool (see section 8.2). The syntax of the script is: export –l <login> -p <password> –u <host>[:<port>][/<database name>] [-i] –f <outputFilename> –s [-P <property name>=<property value>]* where: --login <login>. Login name used to access to the Virtual DataPort server. The user must have sufficient access rights to export data. --password password. Password used to access to the server. --uri <host>:[<port>][/<database name>]. URI of the server (e.g: localhost:9999/admin). If no database is specified in the URI, the script exports all the metadata of the server. Otherwise, it only exports the metadata of that database. If no port is specified, the script tries to connect to the port 9999. -i. Export only WWW wrappers. This option is useful to export content from an ITPilot server, or to export ITPilot wrappers [ITPILOT] from a Virtual DataPort database so that they can be later imported to another ITPilot server. Note: Only use this option if the URI contains a database name. –f outputFilename. Name of the target file. -s. If present, Virtual DataPort servers will switch to single user mode while importing the file outputFilename. -P name=value. Set of settings (explained in section 8.2) that control the export process. You can specify one or more properties following the <property name> = {yes | no} format. The available properties are:

• includejars • includeEnvSpecificElements



• includeNonEnvSpecificElements

• includeDependencies

• includescanners

• includeCustomComponents

• replaceExistingElements

• dropElements Example: export -l admin -p admin -u //localhost:9999/admin -f serverexport.vql –P includejars=yes includescanners=yes includeEnvSpecificElements=no includeNonEnvSpecificElements=yes The script exports all the metadata of the Virtual DataPort server running in localhost, port 9999, using the credentials admin / admin. The result of the export is saved to the file serverexport.vql. The import script imports Virtual DataPort metadata that was obtained using the export utility, to a list of Virtual DataPort servers. The URI of these servers can be provided by command line or readed from a text file. Thus, it is possible to restore a backup copy made with export and to replicate the metadata of a server into a cluster of servers. The syntax of the script is: import [-u] -f path-to-file [-s servers-file | -l server (-l server)*] where: -u (--singleuser). The server switches to single user mode (see Advanced VQL Guide [VQL]) while importing the metadata of the file. Note: you are strongly advised to always import metadata in single user mode. -f path-to-file. Path to the file containing the metadata to be imported. -l server. URI of the server to which the metadata is imported. The syntax of the URI is host[:port]/database?user@password. If port is missing, the script assumes that the port is 9999. -s servers-file. Instead of using the parameter –l, we can provide the path to a file containing URIs of the servers to which the metadata will be imported. Each URI has to be indicated in a separate line, with the same syntax of the parameter –l. Example: import -f export.vql -l localhost/admin?admin@admin1 -l host2:9099/admin?admin@admin2 The script imports the metadata contained in export.vql to the servers running in the local machine, port 9999 and in host host2, port 9099. For both servers, authentication is made against the database admin, and login admin. The passwords for the servers are admin1 and admin2, respectively. import -f export.vql -s servers.conf The script imports the metadata contained in export.vql and reads the target URIs from the file servers.conf.



11.4 CONFIGURING SEVERAL INSTANCES OF A VIRTUAL DATAPORT SERVER

It is possible to launch two or more instances of a Virtual DataPort Server at the same time. Each instance is executed independently, which means that each one is a different process of the operating system. However, all of them share the same metadata (data sources, views, etc) and the same embedded Web container. When executing two or more instances of the same Server, one of the instances is the Master and the others are the Slaves. The Master is the only one that can modify the metadata of the Server (create/edit/remove data sources, views, etc). This instance has to be launched before the Slave instances.

11.4.1 How to Configure the Slave instances

Let us say that we want to add two new Slave instances called “aux_1” and “aux_2”. To do this, follow these steps:

1. Edit the file $DENODO_HOME/conf/vdp/VDBConfiguration.properties and add the following properties:

vdp.instances=aux_1,aux_2 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_1.registryURL=localhost com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_1.registryPort=19999 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_1.shutdownPort=19998 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_1.factoryPort=19997 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_1.odbcPort=19996 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_2.registryURL=localhost com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_2.registryPort=29999 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_2.shutdownPort=29998 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_2.factoryPort=29997 com.denodo.vdb.vdbinterface.server.VDBManagerImpl.aux_2.odbcPort=29996

Note that for each element of the property vdp.instances, you have to add the following properties: com.denodo.vdb.vdbinterface.server.VDBManagerImpl .<name of the instance>.registryURL com.denodo.vdb.vdbinterface.server.VDBManagerImpl .<name of the instance>.registryPort com.denodo.vdb.vdbinterface.server.VDBManagerImpl .<name of the instance>.shutdownPort com.denodo.vdb.vdbinterface.server.VDBManagerImpl .<name of the instance>.factoryPort com.denodo.vdb.vdbinterface.server.VDBManagerImpl .<name of the instance>.odbcPort

2. Optionally, you can configure the options of the JVM for each new instance. To do this, add the following

properties to the file $DENODO_HOME/conf/vdp/VDBConfiguration.properties. java.env.aux_1.DENODO_OPTS_START=-mx2048m -XX\:MaxPermSize\=256m java.env.aux_1.DENODO_OPTS_STOP=-mx64m java.env.aux_2.DENODO_OPTS_START=-mx2048m -XX\:MaxPermSize\=256m java.env.aux_2.DENODO_OPTS_STOP=-mx64m

Note that for each element of the property vdp.instances, you have to add the following properties: java.env.<name of the instance>.DENODO_OPTS_START= java.env.<name of the instance>.DENODO_OPTS_STOP=

The property DENODO_OPTS_START has the JVM options that the instance will launched with. The property DENODO_OPTS_STOP has the JVM options that the script that stops the Server will be launched with.



Note: the characters “:” and “=”have to be escaped with the character “\”.

3. Execute the script $DENODO_HOME/bin/regenerateInstanceScripts. This scripts will create the directory $DENODO_HOME/bin/instances, which will contain the scripts to launch the Slave instances of the Master Server. For each instance, there will be:

a. A script to launch the instance: vqlserver_<name of the instance>_startup

b. A script to stop the instance:

vqlserver_<name of the instance>_shutdown

c. If running the Server in Windows, a script to create a new Windows service, remove it, launch and stop the service: vdpservice_<name of the instance>.bat

4. Launch the Master instance before the Slave instances. You can do it from the Denodo Control Center or

executing the script $DENODO_HOME/bin/vqlserver_startup. 5. Launch the Slave instances by executing the following scripts:

$DENODO_HOME/bin/instances/vqlserver_aux_1_startup $DENODO_HOME/bin/instances/vqlserver_aux_2_startup

Now you can connect to any of the three instances (the Master or the Slaves) to query views. However, you have to connect to the Master instance to modify the metadata of the Server. That is, to create/edit/delete data sources, views, etc.


Appendices 186

12 APPENDICES

12.1 JDBC DRIVERS

The following table contains the JDBC drivers that have been successfully tested with Virtual DataPort 4.6. For each DBMS, the table indicates:

• Class. Class name that must be specified when importing a data source using that driver. • URI. URI format used by the driver.

DBMS Class URI DB2 8.2 DB2 9 com.ibm.db2.jcc.DB2Driver jdbc:db2://<hostName>:<port>/<databaseName>

Informix 7 com.informix.jdbc.IfxDriver jdbc:informix-sqli://<hostName>:<port>/<databaseName>:informixserver=<server>

MySQL 4.0.15 MySQL 4.1.1 MySQL 5.x

com.mysql.jdbc.Driver jdbc:mysql://<hostName>:<port>/<databaseName>

Oracle 8i Oracle 9i Oracle 10g Oracle 11g Oracle EBS 12

oracle.jdbc.OracleDriver jdbc:oracle:<protocol>:@<hostName>:<port>:<databaseName> Protocols: thin (recommended), oci, oci8, kprb

PostgreSQL 7.2.3 PostgreSQL 7.4.6 PostgreSQL 8 PostgreSQL 9

org.postgresql.Driver jdbc:postgresql://<hostName>:<port>/<databaseName>

SQL Server 2000 SQL Server 2005 SQL Server 2008 SQL Server 2008R2

net.sourceforge.jtds.jdbc.Driver jdbc:jtds:sqlserver://<hostName>:<port>/<DatabaseName>

SQL Server 2000 SQL Server 2005 SQL Server 2008 SQL Server 2008R2

com.microsoft.jdbc.sqlserver.SQLServerDriver

jdbc:microsoft:sqlserver://<hostName>;DatabaseName=<databaseName>

Sybase Adaptive Server Enterprise 12.5B Sybase Adaptive Server Enterprise 15

net.sourceforge.jtds.jdbc.Driver jdbc:jtds:sybase://<hostName>:<port>/<DatabaseName>

Sybase Adaptive Server Enterprise 12.5B Sybase Adaptive Server Enterprise 15

com.sybase.jdbc3.jdbc.SybDriver jdbc :sybase:Tds:/<hostName>/:/<port>/<database>

Table 9 JDBC Drivers tested with Virtual DataPort

Note: The following drivers are not included with Virtual DataPort, although they can be downloaded from the websites of those companies: IBM DB2, MySQL, Microsoft SQL Server and Sybase. Any JDBC driver should work correctly with Virtual DataPort, but the ones listed in Table 9 have been successfully tested.


Appendices 187

To use a JDBC driver not included with DataPort, copy the jar file of the driver to $DENODO_HOME/extensions/thirdparty/lib Alternatively, when creating a new JDBC data source, you can directly specify the .jar file containing the desired driver (see section 5.3.1).


Appendices 188

12.2 MULTIDIMENSIONAL TO RELATIONAL MAPPING

Virtual DataPort can extract data from multidimensional databases by executing MDX queries. MDX (Multidimensional Expressions) is a query language for multidimensional databases, as SQL is a query language for relational databases. The result of an MDX query is formed by axes, which are similar to the dimensions of a multidimensional structure. Virtual DataPort maps these axes into a relational structure by creating one field for each value of the ON COLUMNS axis and one field for each dimension of the ON ROWS axis. If a dimension of the ON ROWS axis is hierarchical, there will be a column for each level of this dimension. Note: Virtual DataPort only supports MDX queries with the ONROWS and ONCOLUMNS axes.

Figure 121 Relational star schema

SELECT [Measures].MEMBERS ON COLUMNS, NON EMPTY CROSSJOIN ([Region].MEMBERS, [Time].MEMBERS) ON ROWS FROM [$Sales]

Figure 122 MDX query for Visual result mapping (1)

Table 10 contains the result of executing a multidimensional base view created with the MDX query of Figure 122 and Visual result mapping.

RegionLevel01 RegionLevel02 TimeLevel00 TimeLevel01 units_sold units_returned RegionLevel00 All Region <null> <null> All Year <null> 1600 80 All Region <null> <null> All Year 2010 1000 32 All Region <null> <null> All Year 2009 600 48 All Region Europe <null> All Year <null> 800 40 All Region Europe <null> All Year 2010 500 15 All Region Europe <null> All Year 2009 300 25 All Region Europe Spain All Year <null> 500 25 All Region Europe Spain All Year 2010 250 10 All Region Europe Spain All Year 2009 250 15 All Region Europe France All Year <null> 300 15 All Region Europe France All Year 2010 250 5 All Region Europe France All Year 2009 50 10 All Region America <null> All Year <null> 800 40


Appendices 189

All Region America <null> All Year 2010 500 17 All Region America <null> All Year 2009 300 23 All Region America U.S.A All Year <null> 600 30 All Region America U.S.A All Year 2010 400 15 All Region America U.S.A All Year 2009 200 15 All Region America Brazil All Year <null> 200 10 All Region America Brazil All Year 2010 100 2 All Region America Brazil All Year 2009 100 8

Table 10 Contents of a SAP BW base view with Visual result mapping (1)


Appendices 190

12.3 USING THE ADVANCED CONDITIONS EDITOR

The advanced conditions editor allows creating complex conditions in a purely graphical manner. This process is described below. On the left side of the screen we will find menus for creating various values that can appear as operands in the conditions:

• Constants. This menu allows creating constants of the various data types supported by Virtual DataPort. • Functions. This menu allows creating an invocation to one of the functions permitted by Virtual DataPort.

The functions can receive constants, attributes or the result of evaluating other functions as parameters. They return one result. The list of available functions and examples of use of each of them can be seen in the “Syntax of Condition Functions” section of the VQL Advanced Guide [VQL].

• Attributes. This corresponds to the list of attributes of the view to which the selection is applied. The attributes can act as function parameters and/or appear directly as operands in the conditions.

Figure 123 Editor of selection conditions

On the right of the screen we will find menus to select the various operators that can appear in the conditions:

• Operators that can participate in the simple conditions. These are grouped according to the data type to which they can be applied.

• Logical operators (AND, OR, NOT). These are used to combine the different simple conditions in a Boolean expression.

The center boxes of the screen allow constructing three types of elements from top to bottom: values that appear in the conditions, simple conditions and compound Boolean conditions. The box on the left of each group is a workspace


Appendices 191

for creating new elements, while the box on the right shows the elements already created. The following subsections describe in more detail how each of these types of elements is created. Finally, the “Selection condition” box contains the selection condition created.

12.3.1 Creating Values for the Conditions

To create a new constant value the following actions are required:

1. Select the data type of the constant in the ‘Constants’ drop-down menu on the left side of the screen and click on it or drag&drop it to the work space where values are created (box on top left).

2. The type selected will appear in the workspace together with a text area to fill in the value of the constant. • Where the constant belongs to a simple data type, the value required can be written directly in

the text area. • Where it belongs to a compound type (array or register), a new panel will be opened

on clicking the ‘□’ button alongside the text area to edit the required constant values (see section 12.3.4).

• If it belongs to the xml type, a panel will be opened on clicking the ‘□’ button that allows writing the required value required to be written or loading it from a file.

3. On clicking the ‘>’ button, the new constant will appear in the list of values created (upper right-hand box).

To create a new function-type value the following actions are required:

1. Select the required function in the ‘Functions’ drop-down menu on the left side of the screen and click on it or drag&drop it to the work space for creating values (box on top left).

2. The function selected will appear in the workspace together with an area to fill in the value of each parameter of the function. The values of the parameters should be values present in the list of created values (box on top right). To assign a value already created as a parameter of a function drag&drop the value created to the parameter area. Press the ‘>’ button that appears beside the function, and this will appear in the list of values created (box on top right).

In our example, we will create a new constant value of the type Float. For this we will drag the type Float to the work space and fill in the value of same, in this case it will be 1000 (see Figure 124).


Appendices 192

Figure 124 Creating a constant value

12.3.2 Creating Simple Conditions

To create a new simple condition the following actions are required:

1. Select the required simple condition operator in the drop-down menus on the right side of the screen and click on it or drag&drop it to the work space, where the simple conditions are created (left center box).

2. The operator selected will appear in the work space with an area to fill in its operands. The operands

can be either attributes of the input view (present in the “Fields” drop-down menu of the left side of the screen) or values already created (the list of which is displayed in the box on the top right). To assign an attribute or a value already created as an operand of the condition drag&drop the element to the parameter area. Pressing the ‘>’ button that appears beside the simple condition will cause this to appear in the list of simple conditions already created (centre right box).

In our example, we will select the operator ‘>=’ associated with the type Float and drag&drop same to the work space. We will use the attribute revenue as left operand. We will use the constant value Float(1000) created in the previous section as right operand. See Figure 125.


Appendices 193

Figure 125 Creating a simple condition

12.3.3 Creating Boolean Conditions

To create a new Boolean condition the following actions are required:

1. Click on the required Boolean operator (AND, OR or NOT) in the drop-down menus on the right side of the screen. A Boolean condition will be created in the work space (left lower box).

2. The operands of the selected operator can be conditions previously created (right center box) and other Boolean conditions created beforehand. To assign a condition already created as an operand of the new Boolean condition drag&drop the condition to the operand area, press the ‘>’ button that appears beside the Boolean condition, this then appears in the list of Boolean conditions already created (right lower box).

3. Drag&drop the condition to be added to the selection to the “Selection Condition” box. 4. Press ‘ok’ to return to the selection view creation page, where the new condition will be displayed.

In our example, we will not create compound Boolean conditions, so we will directly drag&drop from the simple condition created in the previous section to the “Selection Condition” box.

12.3.4 Creating Conditions that Use Compound-type Constants

Denodo Virtual DataPort supports the modeling of data types with tree structure using register and array types (see Advanced VQL Guide [VQL]). It is possible to create constants of these compound types and to use them in the selection conditions. An explanation of how to do so will be given in this section through an example.


Appendices 194

Figure 126 RevenueSum schema

In the example, a base relation called RevenueSum will be used (see Figure 126). RevenueSum can be created as follows:

1. Using the Web Service datasource sales (see section 5.3.5), create a base view based on the getSumRevenueByTaxIds operation. This operation has an input parameter which is an array of strings (the taxId of each client); and returns the sum of the revenue of all specified clients (float-type value).

2. The new base relation will be called RevenueSum. The name of its attributes will be changed to

clients and totalrevenue, respectively. The clients attribute is an ‘array of registers’. More specifically, DataPort automatically generates an array type called getsumrevenuebyids_in0. Each element in this array will be a register, which is also automatically generated by DataPort and called getsumrevenuebyids_in0_string. The elements in this last type have a single text-type field and they will be renamed to ‘taxId’.


Appendices 195

Figure 127 Compound values editor

In this example, we will create a selection view using a condition on RevenueSum to obtain the total obtained by adding the revenues of the clients with taxIds ‘B78596011’ and ‘B78596012’. The steps to follow are:

1. Select the option ‘Combine – Toolbox – Projection/Selection’. 2. Drag RevenueSum to the work area to create a new selection view. 3. Click the ‘Set Condition’ option to open the selection conditions editor.

The compound data type getSumRevenuebyTaxIds_IN0 will now appear in the ‘Constants’ drop-down on the left of the screen, along with the single data types.

4. Drag & drop the getSumRevenuebyTaxIds_IN0 type to the value creation area and click the ‘□’

button. A new pop-up will appear as shown in Figure 127 to create a new getSumRevenuebyTaxIds_IN0 type constant.

5. Click twice the ‘+’ button beside ‘Value’. We need two elements of the

getSumRevenuebyTaxIds_IN0 type because the array must have two elements.

6. Click on the new elements to expand them (Figure 128) and edit the values of each array element (they will always be of a register type).


Appendices 196

7. Enter the value ‘B78596011’ in the field taxId of the first register and ‘B78596012’ in the same field of

the second register.

Figure 128 Creating a value of type getSumRevenuebyTaxIds_IN0

8. Click ‘Ok’ to close the popup. The new compound value will appear in the creation area. 9. Click the ‘>’ button to create the new constant (it is represented in VQL by

{ROW(B78596011),ROW(B78596012)}). See “Conditions with Compound Values” in the Advanced VQL Guide [VQL] for more information on ROW and {} constructors.

The new constant can now be used to create the query condition required on the clients field. The creation process for this condition is the same as with single-type constants (see section 12.3.2). Figure 129 shows the condition created in the ‘Selection Condition’ box (represented in VQL by: clients={ROW(B78596011),ROW(B78596012)}. Figure 130 shows the result of running the resulting selection view (see section 5.4.9 to obtain information about how to execute views and navigate through the obtained results when these are of compound type).


Appendices 197

Figure 129 Condition using compound-type values

Figure 130 Result of executing the view RevenueSum


Appendices 198

12.4 CONFIGURING LOGS

Virtual DataPort uses the Jakarta-log4j [LOG4J] log system. This system allows log messages to be controlled with different levels of granularity and is configurable through configuration files. Some knowledge of log4j is required to modify the log configuration file. Briefly, Log4j allows defining a series of categories on which the log level will be specified independently. Log4j defines different log levels: DEBUG, INFO, WARN, ERROR and FATAL. NOTE: In production environments, we recommend using an ERROR log level. Lower levels degrade the system performance and are only recommended for debugging tasks. The Virtual DataPort Server logs are stored in the path DENODO_HOME/logs/vdp, where DENODO_HOME is the base installation path. The Administration tool logs are stored in the path DENODO_HOME/logs/vdp-admin. The log4j.xml configuration file of the Virtual DataPort Server logs can be found in the path DENODO_HOME/conf/vdp. Among other options, it is possible to modify the path where logs are stored and the log level of each of the categories defined in the application. See Log4J [LOG4J] documentation for more information. The log4j.xml configuration file of the Administration tool can be found in the path DENODO_HOME/conf/vdp-admin.


Appendices 199

12.5 INSTALLING THE OPENAJAX MASHUP EDITOR

Follow these steps to install the OpenAjax Mashup Editor.

1. Install an HTTP Server such as the Apache HTTP Server (http://httpd.apache.org/)

2. Install PHP (http://php.net/downloads.php) with the module ‘Sqlite PDO’ and the ‘ZIP’ extension.

• Open the $PHP_HOME/php.ini and uncomment or add the following lines : [PHP_PDO] extension=php_pdo.dll [PHP_PDO_SQLITE] extension=php_pdo_sqlite.dll [PHP_ZIP] extension=php_zip.dll

3. Download the following Subversion projects from these URLs: • https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/gadgets/tags/openajax.org-

20081023 • https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/hub20/tags/openajax.org-

20081023 • https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/repository/tags/openajax.org-

20081023 Or download them from the Web interface provided by this repository: • http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/gadgets/tags/openajax.org-

20081023/ • http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/hub20/tags/openajax.org-

20081023/ • http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/repository/tags/openajax.org-

20081023/ These URLs belong to the tag openajax.org-20081023 of the gadgets, hub20 and repository subprojects of the OpenAjax Subversion repository.

4. Copy the downloaded projects to a folder of your HTTP server, i.e. $APACHE_HOME/htdocs/mashupapp. The Mashup Editor expects these directories to be located alongside: • mashupapp/gadgets/ • mashupapp/hub11/ • mashupapp/repository/

Important: remove all the directories named ‘.svn’ within these directories, because their content could interfere with the deployment of some widgets.

5. Rename the files

mashupapp/gadgets/config/config.php.sample to config.php. and mashupapp/gadgets/config/subdomains.cfg.sample to subdomains.cfg

6. Edit the file mashupapp/gadgets/config.php:

• Change the value of SHINDIG_BASE_URL to http://<hostname>/mashupapp/gadgets/

http://httpd.apache.org/

http://php.net/downloads.php

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/gadgets/tags/openajax.org-20081023

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/gadgets/tags/openajax.org-20081023

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/hub20/tags/openajax.org-20081023

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/hub20/tags/openajax.org-20081023

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/repository/tags/openajax.org-20081023

https://openajaxallianc.svn.sourceforge.net/svnroot/openajaxallianc/repository/tags/openajax.org-20081023

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/gadgets/tags/openajax.org-20081023/

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/gadgets/tags/openajax.org-20081023/

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/hub20/tags/openajax.org-20081023/

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/hub20/tags/openajax.org-20081023/

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/repository/tags/openajax.org-20081023/

http://openajaxallianc.svn.sourceforge.net/viewvc/openajaxallianc/repository/tags/openajax.org-20081023/


Appendices 200

• Change the value of REPOSITORY_URL to http://<hostname>/mashupapp/repository/

7. Edit the file mashupapp/gadgets/config/subdomains.cfg:

• Replace every line cXX.openajax.org with the name of the server where the Mashup editor is deployed (i.e. if the host name is openajax.denodo.com, replace c1.openajax.org with openajax.denodo.com, c2.openajax.org with openajax.denodo.com, …)

8. Rename the file

mashupapp/repository/config/config.php.sample to config.php

9. Edit the file mashupapp/repository/config.php: • Change the value of REPOSITORY_URL to

http://<server’s IP address>/mashupapp/repository/

10. Access to http://<server’s IP address>/mashupapp/repository/install.php and click on the ‘Start installation’ button to initialize the OpenAjax widgets database.

If the last message is not ‘Install finished!’ it means that the install process didn’t finish correctly. Usually, the reason for this is that the request “timed out”. To solve this, change the value of the property max_execution_time to 60 in the php.ini file of your PHP installation and try again. Also remember that you should always use the same hostname in all the previous steps. Now the OpenAjax Mashup Editor is installed at http://<server’s IP address>/www/gadgets/samples/newmashup.php and the Repository Browser, at http:// <server’s IP address>/mashupapp/repository/scaffold.php.


Appendices 201

12.6 INSTALLING THE DENODO SOLUTION FOR MICROSOFT SHAREPOINT

The Denodo Solution for Microsoft SharePoint is used by the Web Parts widgets published with DataPort (see section 7.6). It contains a DLL that, among other things, will connect to the auxiliary Web Service used by those widgets to obtain the data from a specific view. Important: you only need to deploy this solution the first time you want to deploy a Web Part. Follow these steps to deploy the solution:

1. Copy the file $DENODO_HOME/webapps/webpart-core/DenodoWebPart-4.6.0.wsp to your SharePoint server.

2. Execute the following commands in your SharePoint server console:

a. cd C:\Program Files\Common Files\Microsoft Shared\web

server extensions\12\bin

b. stsadm.exe -o addsolution –filename "DenodoWebPart-4.6.0.wsp"

c. stsadm.exe -o deploysolution -name "DenodoWebPart-4.6.0.wsp" -immediate -allcontenturls -allowCasPolicies

The allowCasPolicies parameter is necessary because the solution contains Code Access Security (CAS) policies, to allow connections to an external server (auxiliary Web Service), write log information to files, read the XML configuration file of each Web Part, etc.

d. stsadm.exe -o execadmsvcjobs

you can omit this step if the “Microsoft SharePoint Timer” service is enabled. To list the existing solutions, execute: stsadm.exe -o enumsolutions To undeploy and delete the solution execute these commands in your SharePoint server console:

1. stsadm.exe -o retractsolution -name " DenodoWebPart-4.6.0.wsp" -immediate -allcontenturls

2. stsadm.exe -o execadmsvcjobs

you can omit this step if the “Microsoft SharePoint Timer” service is enabled.

3. stsadm.exe -o deletesolution -name " DenodoWebPart-4.6.0.wsp"


Appendices 202

12.7 TRANSFORMING INCOMING/OUTGOING SOAP MESSAGES WITH XSLT STYLESHEETS

As explained in section 6.2.3, we can define XSLT stylesheets [XSLT] to transform incoming/outgoing SOAP messages. Thus, we avoid any modifications to existing SOAP clients. This section explains a possible way to generate these XSLT stylesheets more easily leveraging on some popular open-source tools. Before starting, we need to obtain some tools that will help us to define these transformations:

• SoapUI. (http://www.soapui.org) • Eclipse Web Tools. (http://www.eclipse.org/webtools). Set of Eclipse plugins that includes editors to

create and test XSLT transformations. Specifically, we will use the “XML Editor Tools” and the “XSL Developer Tools”.

First, we have to publish a new SOAP Web service from Virtual DataPort (section 6 explains how to do it). Make the structure of the new Web service as similar as possible to the existing Web service. By doing this, the XSLT stylesheets will be simpler:

• The operations of the new Web service should have the same parameters as the existing one. • Change the names of operations and parameters so they match with the names of the existing Web

service. • You can also use FLAT / NEST operations (see section 5.4.5) to adjust the hierarchical structure of the

published web service to thet of the target web service. For each operation, we need:

• For the “Input XSLT Transformation”: o A SOAP request sent by the existing Web service client. o A SOAP request expected by the new Web service.

• For the “Output XSLT Transformation”:

o A SOAP response expected by the existing Web service client. o A SOAP response sent by the new Web service.

To obtain these sample messages, we need the WSDL documents [WSDL] of the existing and the new Web services. If we have deployed the new Web service in the embedded Web container, we can obtain its WSDL at http://localhost:9090/server/<VDP database name>/<name of service>/services/<name of service>?wsdl For both WSDL files, we have to obtain a sample SOAP request and a sample SOAP response. We can use SoapUI to do this:

1. Launch SoapUI. 2. Create a new project for each WSDL. In the “New Project” dialog, select the options “Create

Requests” and “Create MockService”. After creating the project, the left-side panel contains a list of the operations of the Web service.

3. For each operation, double-click on “Request” to open the template of the request. 4. For each operation, double-click on “Response” to open the template of the response expected by the

client. 5. Save each template: Right-click on each template’s window and then, on “Save as…”

After this, we should have four SOAP files for each operation:

a) A request sent by the existing Web service client. b) A response sent by the existing Web service. c) A request expected by the new Web service

http://www.soapui.org/

http://www.eclipse.org/webtools


Appendices 203

d) A response sent by the new Web service. Now, for each operation, we have to generate two XSLT stylesheets:

a) One XSLT stylesheet to transform the request sent by the existing client into what the new Web service expects.

b) Another XSLT stylesheet to transform the response sent by the new Web service into what the existing client expects.

There are many XML/XSLT editors to write XSLT transformations but in this appendix, we focus on “XML Editor Tools” and “XSL Developer Tools” from the Eclipse Web Tools. We can load the initial SOAP messages into the editor and create the XSLT stylesheets using some useful features of these editors: syntax highlighting and transformation execution/debugging. Once the XSLT transformations are finished, we have to execute them and validate their output against the schema defined in the WSDLs. There are two ways to validate the output of XSLT transformations:

1. Using Eclipse: extract the schema of the message from the WSDL and paste it in an .xsd file. Then, using the XML editor, execute the XSD validation of the transformed XML (just the content inside the <Body> tag)

2. Using SoapUI: in the appropriate window, replace the old XML with the result of the

transformation.Then, open the contextual menu and click on 'Validate'. Important note: if the schema of a Web service changes, their XSLT stylesheets have to be modified accordingly. Changes in the schema of a Web service include:

a) Renaming existing operations. b) Removing operations.

c) Adding / renaming / removing parameters of existing operations.

In addition, if we add an operation to an existing Web service, its XSLT transformation will be empty.


Appendices 204

12.8 JMS CONNECTION DETAILS: JNDI PROPERTIES AND CLIENT JARS

This appendix provides information about how to connect Denodo with some of the most popular JMS servers:

• Configuration values required to configure a Denodo Web service or a JMS listener, so they can connect to a JMS server.

• List of the client jars required by the Denodo Web services and the JMS listeners to connect to the most

popular JMS servers. The clients of JMS servers require these jars because the process of establishing the connection relies on proprietary classes of the vendor.

Note: the first time a Web service is deployed into the embedded Web container of Virtual DataPort, copy these client jars into the $DENODO_HOME/resources/apache-tomcat/common/lib and restart the Virtual DataPort server (we only have to copy them once). If the Web service is deployed in an external application server, copy these jars into the /WEB-INF/lib directory of the war file, before deploying it. Note: the first time we create a JMS listener, we have to copy the client jars of the JMS vendor to the directory $DENODO_HOME/extensions/thirdparty/lib and restart the Virtual DataPort server.

12.8.1 Apache ActiveMQ

The client jars for Apache ActiveMQ are located in $ACTIVE_MQ_HOME/lib: • activemq-core.jar • geronimo-j2ee-management.jar • geronimo-jms_1.1_spec.jar

The values for a default installation are:

Property Property value JNDI Provider URL tcp://acme:8161 Broker URL tcp://acme:61616

Table 11 Default values for Apache Active MQ

12.8.2 IBM WebSphere MQ

The client jars for IBM WebSphere MQ are located in $WEBSPHERE_MQ_HOME/java/lib: • com.ibm.mq.jar • com.ibm.mqjms.jar • com.ibm.mq.jmqi.jar • dhbcore.jar • jta.jar • fscontext.jar • providerutil.jar

The properties for a default installation are:


Appendices 205

Property Property value Host acme Port 1414

Channel SYSTEM.DEF.SVRCONN

Queue Manager QM_acme

Table 12 Default values for IBM WebSphere MQ

12.8.3 OpenJMS

The client jars for OpenJMS are located in $OPEN_JMS_HOME/lib: • openjms-client.jar • exolabcore.jar

The JNDI properties for a default installation are:

JNDI property name Property value java.naming.provider.url rmi://acme:1099

java.naming.factory.initial org.exolab.jms.jndi.InitialContextFactory

transport.jms.ConnectionFactoryJNDIName

To subscribe to queues: JmsQueueConnectionFactory To subscribe to topics: JmsTopicConnectionFactory

Table 13 Default JNDI properties for OpenJMS

Note: The list of jars and JNDI properties are valid for OpenJMS 0.7.6.1. They may change in future versions of OpenJMS.

12.8.4 Progress SonicMQ

The client jars are located in $SONIC_MQ\Sonic\MQ8.0\lib: • mfcontext.jar • sonic_Client.jar • sonic_Crypto.jar • sonic_XA.jar • sonic_XMessage.jar

The JNDI properties for a default installation are:


Appendices 206

JNDI property name Property value java.naming.provider.url acme:2506

java.naming.factory.initial com.sonicsw.jndi.mfcontext.MFContextFactory

transport.jms.ConnectionFactoryJNDIName

ConnectionFactory Note: This connection factory name has to be defined in the JNDI service of SonicMQ.

Table 14 Default JNDI properties for Progress SonicMQ


Appendices 207

12.9 INSTALLING THE CONNECTORS FOR SAP ERP, SAP BW AND SAP BI

To use BAPI (SAP ERP) data sources or Multidimensional data sources with SAP BW and SAP BI, we need to install their connectors in the system where Virtual DataPort runs.

12.9.1 Installing the Connector for SAP ERP (BAPI Data Sources)

Follow these steps to install the SAP ERP connector:

1. Obtain the SAP Java Connector 3 (SAP JCo). It can be downloaded from http://service.sap.com/connectors, section SAP Java Connector -> Tools & Services. If Virtual DataPort runs on a 32-bit O.S, download the 32-bit connector. If Virtual DataPort runs on a 32-bit JVM, on a 64-bit O.S, download the 32-bit connector. If Virtual DataPort runs on a 64-bit JVM, download the 64-bit connector.

2. Uncompress the downloaded package.

3. Copy the file sapjco3.jar to the directory $DENODO_HOME/extensions/thirdparty/lib

4. In Linux: copy the file libsapjco3.so to the directories $DENODO_HOME/jre/lib/i386 and

$DENODO_HOME/jre/lib/i386/client. If Virtual DataPort is configured to use the JVM not included with the Denodo Platform, copy this file to the directories jre/lib/i386 and /jre/lib/i386/client of the JVM that runs Virtual DataPort.

5. In Windows:

a. Copy the file sapjco3.dll to the directory $DENODO_HOME/extensions/thirdparty/lib

b. Add the directory $DENODO_HOME/extensions/thirdparty/lib to the environment

variable PATH.

c. According to SAP note #684106 (https://service.sap.com/sap/support/notes/684106), if Virtual DataPort and SAP ERP are NOT running in the same host, we need to install the latest Visual Studio 2005 runtime libraries:

i. Browse to http://www.microsoft.com/technet/security/bulletin/MS09-035.mspx ii. Scroll to section ‘Affected Software’. iii. Click on the link ‘Microsoft Visual C++ 2005 Service Pack 1 Redistributable Package’

(KB973544) iv. Download one of the following platform-specific files depending on your scenario:

1. Virtual DataPort running on a 32-bit O.S: vcredist_x86.exe 2. Virtual DataPort running on a 32-bit JVM, on a 64-bit O.S: vcredist_x86.exe 3. Virtual DataPort running on a 64-bit JVM: vcredist_x64.exe 4. Itanium system: vcredist_IA64.exe

v. Execute the downloaded file and follow the installation instructions.

6. To check that the JCo connector is working, execute java -jar "$DENODO_HOME/extensions/thirdparty/lib/sapjco3.jar" You should see something like Figure 131.

http://service.sap.com/connectors

https://service.sap.com/sap/support/notes/684106

http://www.microsoft.com/technet/security/bulletin/MS09-035.mspx


Appendices 208

Figure 131 SAP Java Connector (JCo) test screen

7. Restart the Virtual DataPort server and the Administration Tool.

12.9.2 Installing the Connector for SAP BW and SAP BI (Multidimensional Sources)

Follow these steps to install the SAP BW and SAP BI connector:

1. Obtain the SAP Business Intelligence Java Software Development Kit (BI Java SDK) distributed with SAP NetWeaver.

2. Uncompress the package and copy all its jars, including the jars in its subdirectories, to

$DENODO_HOME/extensions/thirdparty/lib

3. Restart the Virtual DataPort server and the Administration Tool.


References 209

REFERENCES

[ARCN] Denodo Aracne Administration Guide. Denodo Technologies.

[AXIS] Apache Axis. Apache Software Foundation. http://ws.apache.org/axis/

[DENINST] Denodo Platform Installation Guide. Denodo Technologies.

[DERBY] Apache Derby. http://db.apache.org/derby/

[DEV] Denodo Virtual DataPort Developer Guide. Denodo Technologies.

[GENITP] Denodo ITPilot Generation Environment. Denodo Technologies.

[GMINI] Google Mini. http://www.google.com/enterprise/mini/

[GSEARCHLANG] Lenguajes soportados por Google Mini. http://code.google.com/enterprise/documentation/xml_reference.html - request_subcollections_auto

[HTTP-AUTH] HTTP Authentication: Basic and Digest Access Authentication. http://www.ietf.org/rfc/rfc2617.txt

[ITPILOT] Denodo ITPilot User Guide. Denodo Technologies.

[JCA] Java Cryptography Architecture (JCA) Reference Guide http://download.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html

[JCONSOLE] Using JConsole. http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html

[JMANAGE] jManage: Open Source Application Management. http://www.jmanage.org/

[JMS] Java Message Service (JMS). http://java.sun.com/products/jms/

[JMX] Java Management eXtensions (JMX). http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/

[JREG] JAVA Regular Expressions. http://java.sun.com/javase/6/docs/api/java/util/regex/Pattern.html

[JSON] JavaScript Object Notation. http://www.json.org/

[JSR-168] Java Portlet Specification. Version 1.0. http://jcp.org/en/jsr/detail?id=168

[JSR-286] Java Portlet Specification. Version 2.0. http://jcp.org/en/jsr/detail?id=286

[KALV] KeepAlived. Health Checking for LVS & High Availability. http://www.keepalived.org/

[KEYTOOL] Java key and Certificate Management Tool http://java.sun.com/javase/6/docs/technotes/tools/windows/keytool.html or http://java.sun.com/javase/6/docs/technotes/tools/solaris/keytool.html

[LOG4J] The Log4j Project. Apache Software Foundation. http://logging.apache.org/log4j/docs/

[LVS] Linux Virtual Server Architecture. http://www.linuxvirtualserver.org/

[MS-AD] Microsoft Windows Server Active Directory. http://www.microsoft.com/windowsserver2003/technologies/directory/activedirectory/default.mspx, http://www.microsoft.com/windowsserver2008/en/us/ad-main.aspx

[MS-NLMP] NT LAN Manager (NTLM) Authentication Protocol Specification v1 and v2. http://msdn.microsoft.com/en-us/library/cc236621(PROT.10).aspx

[MS_SQL] Microsoft SQL Server. http://www.microsoft.com/sqlserver/

[MYSQL] MySQL Open Source Database. http://www.mysql.com/

[NSEQL] Denodo ITPilot NSEQL Guide. Denodo Technologies.

http://ws.apache.org/axis/

http://db.apache.org/derby/

http://www.google.com/enterprise/mini/

http://code.google.com/enterprise/documentation/xml_reference.html#request_subcollections_auto

http://www.ietf.org/rfc/rfc2617.txt

http://download.oracle.com/javase/6/docs/technotes/guides/security/crypto/CryptoSpec.html

http://java.sun.com/j2se/1.5.0/docs/guide/management/jconsole.html

http://www.jmanage.org/

http://java.sun.com/products/jms/

http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/

http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/

http://java.sun.com/javase/6/docs/api/java/util/regex/Pattern.html

http://www.json.org/

http://jcp.org/en/jsr/detail?id=168

http://jcp.org/en/jsr/detail?id=286

http://www.keepalived.org/

http://java.sun.com/javase/6/docs/technotes/tools/windows/keytool.html

http://java.sun.com/javase/6/docs/technotes/tools/solaris/keytool.html

http://logging.apache.org/log4j/docs/

http://www.linuxvirtualserver.org/

http://www.microsoft.com/windowsserver2003/technologies/directory/activedirectory/default.mspx

http://www.microsoft.com/windowsserver2008/en/us/ad-main.aspx

http://msdn.microsoft.com/en-us/library/cc236621(PROT.10).aspx

http://www.microsoft.com/sqlserver/

http://www.mysql.com/


References 210

[OPENAJAX] OpenAjax Mashup Editor. http://www.openajax.org/, http://sourceforge.net/projects/openajaxallianc

[ORCL] Oracle Database. Oracle Corporation. http://www.oracle.com/database/index.html

[POSQL] PostgreSQL Open Source Database. http://www.postgresql.org/

[RSS] Really Simple Syndication Format (RSS 2.0). http://cyber.law.harvard.edu/rss/rss.html

[SCHED] Denodo Scheduler Administration Guide. Denodo Technologies.

[SHAREPOINT] Microsoft Office SharePoint Server. http://sharepoint.microsoft.com/

[SOAP] SOAP. http://www.w3.org/TR/soap/

[SOAP_JMS] SOAP over Java Message Service (JMS). http://www.w3.org/TR/soapjms/

[TOM] Apache Jakarta Tomcat. The Apache Software Foundation. http://tomcat.apache.org/

[VQL] Denodo Virtual DataPort VQL Advanced Guide. Denodo Technologies.

[WIND] Windows Operating System Family. Microsoft Corporation. http://www.microsoft.com/windows/default.mspx

[WNLB] Network Load Balancing with Windows 2000 Advanced Server http://www.microsoft.com/technet/prodtechnol/windows2000serv/support/nlbcontb.mspx

[WSDL] Web Services Description Language (WSDL). http://www.w3.org/TR/wsdl

[WSS] Web Services Security (WS-Security). http://www.oasis-open.org/committees/wss/

[WSS-UT] Web Services Security Username Token Profile 1.1. http://docs.oasis-open.org/wss/v1.1/

[XA] X/Open Company Ltd. Distributed Transaction Processing: The XA Specification. The Open Group, February 1992.

[XPATH] Xpath Language. http://www.w3.org/TR/xpath

[XSLT] XSL Transformations (XSLT). http://www.w3.org/TR/xslt

http://www.openajax.org/

http://sourceforge.net/projects/openajaxallianc

http://www.oracle.com/database/index.html

http://www.postgresql.org/

http://cyber.law.harvard.edu/rss/rss.html

http://sharepoint.microsoft.com/

http://www.w3.org/TR/soap/

http://www.w3.org/TR/soapjms/

http://tomcat.apache.org/

http://www.microsoft.com/windows/default.mspx

http://www.microsoft.com/technet/prodtechnol/windows2000serv/support/nlbcontb.mspx

http://www.w3.org/TR/wsdl

http://www.oasis-open.org/committees/wss/

http://docs.oasis-open.org/wss/v1.1/

http://www.w3.org/TR/xpath

http://www.w3.org/TR/xslt