Using PowerExchange for Cassandra to Read from and Write ... Library/1/0981...The Cassandra database supports the list, set, and map collection types. You can use PowerExchange ®

Using PowerExchange for Cassandra to Read

from or Write to Cassandra Collections

© Copyright Informatica LLC 2014, 2017. Informatica, the Informatica logo, PowerCenter, and PowerExchange are trademarks or registered trademarks of Informatica LLC in the United States and many jurisdictions throughout the world. A current list of Informatica trademarks is available on the web at https://www.informatica.com/trademarks.html

AbstractThe Cassandra database supports the list, set, and map collection types. You can use PowerExchange® for Cassandra to create virtual tables to normalize data in Cassandra collections, such as a list, set, or map. Use virtual tables in a mapping to perform read and write operations on Cassandra collections. This article explains how to use PowerExchange for Cassandra to read from or write to Cassandra collections.

Supported Versions• PowerExchange for Cassandra for PowerCenter 9.6.1 HotFix 1, 10.x

Table of ContentsCassandra Collections Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

Introduction to Virtual Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Example: List Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3

Example: Set Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Example: Map Collection. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Before You Begin. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4

Reading from a Cassandra List Collection Type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Importing a Cassandra List Source Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Creating a Cassandra List Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5

Writing to a Cassandra List Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Importing a Cassandra List Source and Target Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6

Creating a Cassandra List Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

Reading from a Cassandra Set Collection Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Importing a Cassandra Set Source Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

Creating a Cassandra Set Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

Writing to a Cassandra Set Collection Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Importing a Cassandra Set Source and Target Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11

Creating a Cassandra Set Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

Reading from a Cassandra Map Collection Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Importing a Cassandra Map Source Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

Creating a Cassandra Map Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

Writing to a Cassandra Map Collection Type. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Importing a Cassandra Map Source and Target Definition. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15

Creating a Cassandra Map Mapping. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16

Cassandra Collections OverviewYou can read, write, and manipulate a group of data by using collections in Cassandra.

The Cassandra database supports the following collection types:

2

List

A list is a collection of one or more ordered elements.

Set

A set is a collection of one or more elements.

Map

A map is a collection of key-value pairs.

The Designer imports a column based on the schema that you set for the column family. If a column contains a collection such as a set, list, or map, the Designer creates a virtual table and imports the collection as columns at the same level as other columns. The Designer creates a virtual table for each collection in the column family.

Introduction to Virtual TablesThe Informatica Cassandra ODBC driver creates virtual tables if the column family contains set, list, or map collections.

Virtual tables depict the normalized view of Cassandra collections. You can import virtual tables as an ODBC source definition and create mappings. The virtual table for a collection uses the following naming convention by default: <original column family name>_vt_<original collection name>.

Each virtual table has a key column that references the primary key column in the original column family.

The virtual table has an index column that shows the position of the data within the original array. The index column uses the following naming convention by default: <original column name>_index. Other columns in the virtual table represent the elements in the array and are named after the array element. If the array is of scalar type, the data column uses the following naming convention by default: <original column name>_value.

Example: List CollectionFor example, consider the table list_only_src.

The following CQL statement defines a Cassandra column family that contains a list collection:

create table list_only_src(col_uuid uuid primary key, col_list list<timestamp>);

When you use the Informatica Cassandra ODBC driver to import the column family, the driver creates the list_only_src_vt_col_list virtual table for the list_only_src table.

The following table shows the renormalised view of the list_only_src_vt_col_list virtual table:

col_uuid col_list_index col_list_value

29afe001-c5c5-11e6-822d-e759170e7fa7 0 12/19/2016 08:28:48.0000

29afe001-c5c5-11e6-822d-e759170e7fa7 1 12/19/2016 08:28:48.0000

29b29f20-c5c5-11e6-822d-e759170e7fa7 0 12/19/2016 08:28:48.47.951000000

29ade431-c5c5-11e6-822d-e759170e7fa7 0 12/19/2016 08:28:48.47.951000000

3

Example: Set CollectionThe following CQL statement defines a Cassandra column family that contains a set collection:

create table set_only_src(col_uuid uuid primary key, col_set set<blob>);

When you use the Informatica Cassandra ODBC driver to import the column family, the driver creates the set_only_src_vt_col_set virtual table for the set_only_src table.

The following table shows the renormalised view of the set_only_src_vt_col_set virtual table:

col_uuid col_set_value

29bead10-c5c5-11e6-822d-e759170e7fa7 0x37313336343138373377

29bead10-c5c5-11e6-822d-e759170e7fa7 0x6275646967667277343236353734363537343266

29c7fbe1-c5c5-11e6-822d-e759170e7fa7 0x37313336343138373377 0x393837387572656a64

29c7fbe1-c5c5-11e6-822d-e759170e7fa7 0x393837387572656a64

Example: Map CollectionThe following CQL statement defines a Cassandra column family that contains a set collection:

create table map_only_src(col_uuid uuid primary key, col_map map<inet, text>);

When you use the Informatica Cassandra ODBC driver to import the column family, the driver creates the map_only_src_vt_col_map virtual table for the map_only_ src table.

The following table shows the renormalised view of the map_only_src_vt_col_map virtual table:

col_uuid col_map_key col_map_value

29d939f1-c5c5-11e6-822d-e759170e7fa7 10.67.200.190 String1

29d939f1-c5c5-11e6-822d-e759170e7fa7 fe80::6ce8:9642:8174:1d13 String2

29cd7a21-c5c5-11e6-822d-e759170e7fa7 10.65.200.190 String1

29d123a1-c5c5-11e6-822d-e759170e7fa7 fe80::6ce8:1111:8174:1d13 String2

Before You BeginBefore you use PowerExchange for Cassandra for the tasks in this document, perform the following tasks:

• Install or upgrade PowerCenter®.

• Ensure that you have the PowerExchange for Cassandra license file. You do not require a separate ODBC license to use PowerExchange for Cassandra.

4

Reading from a Cassandra List Collection TypeYou can import a Cassandra column family as an ODBC source definition in the Designer tool and create a mapping that reads from a Cassandra list collection.

To read data from a Cassandra list collection type, perform the following tasks:

1. Import a Cassandra list source definition.

2. Create a Cassandra list mapping and run a workflow to extract data from a Cassandra list collection and load data to a flat file.

Importing a Cassandra List Source DefinitionYou can import a Cassandra column family as an ODBC source definition in the Designer. Cassandra ODBC source definitions represent metadata for Cassandra source column families. After you import the Cassandra source definitions, you can use the source definition in a mapping that reads from Cassandra.

1. In the Designer, click Tools > Source Analyzer to open the Source Analyzer.

2. Click Sources > Import from Database.

3. From the ODBC data source list, select the ODBC data source that you created to access the source tables.

4. Enter the user name and password to connect to the database.

5. Click Connect.

6. In the Select tables list, double-click the database owner heading and then expand the TABLES folder.

7. Select the source table and the virtual table corresponding to the list collection type that you want to import.

8. Click OK to import the source definitions.

The following image shows the list source definitions imported into the Source Analyzer workspace:

Creating a Cassandra List MappingCreate a mapping with the imported Cassandra list to read data from Cassandra and include a flat file target definition so that other business intelligence tools can consume the data.

1. In the Mapping Designer, click Mappings > Create.

The Mapping Name dialog box appears.

2. Select the imported Cassandra source definition.

3. Select a flat file target definition.

4. Click OK.

5. Link the source definition to the target definition.

5

The following image shows the virtual table list source definition, the Source Qualifier transformation, and the flat file target definition:

The following image shows the data preview of the read operation:

6. In the Workflow Manager, create a session and workflow to extract data from the Cassandra list collection and load data to a flat file.

Writing to a Cassandra List TypeYou can import a Cassandra column family as an ODBC target definition and create mappings in the Mapping Designer to write data to a Cassandra list collection.

When you run a Cassandra mapping, the PowerCenter Integration Service uses the Informatica Cassandra ODBC data source to load data to the Cassandra database.

To write data to a Cassandra list collection type, perform the following tasks:

1. Import a Cassandra list source definition.

2. Import a Cassandra list target definition.

3. Create a Cassandra list mapping and run a workflow to extract data from a Cassandra list and load data to a Cassandra list.

Importing a Cassandra List Source and Target DefinitionYou can import a Cassandra column family as ODBC source and target definitions in the Designer. Cassandra ODBC source and target definitions represent metadata for Cassandra source and target column families. After you import the Cassandra target definition, you can use the target definition in a mapping that writes to Cassandra.





5. Click Connect.


7. Select the source table and the virtual table corresponding to the list collection type that you want to import.

6


The following image shows the Source Analyzer with the list source definitions imported into the Source Analyzer workspace:

9. Click Tools > Target Designer to open the Target Designer.

10. Click Targets > Import from Database.

11. From the ODBC data source list, select the ODBC data source that you created to access the target tables.


13. Click Connect.


15. Select the target table and the virtual table corresponding to the list collection type that you want to import.

16. Click OK to import the target definitions.

The following image shows the Target Designer with the target definitions imported into the Target Designer workspace:

Creating a Cassandra List MappingCreate a mapping with the imported Cassandra list source and Cassandra list target to write data to Cassandra.




3. Select the imported Cassandra target definition.

4. Click OK.


For example, consider the mapping source as list_only_src_vt_col_list table and mapping target as list_only_tgt_vt_col_list table in a Cassandra list mapping.

7

The following mapping shows the write operation that you can perform:

You can perform the following write operations on a Cassandra list:

Insert Operation

You can use the insert operation to add one or more rows to a virtual table.

The following image shows the data preview of the target virtual table:

Update Operation

You can use a DD_UPDATE constant in an Update Strategy transformation to update rows in a virtual table.

The following image shows a data preview of the rows in the target table before you update rows in the target table:

The following image shows a data preview of the rows updated in the target virtual table:

Append Operation

You can append rows only to virtual tables created for list type. You must set the list index Update Strategy to -1.

8

The following image shows a data preview of the rows in the target table before you append rows to the target table:

The following image shows a data preview of the rows appended in the target virtual table:

Delete Operation

You can use a DD_DELETE constant in an Update Strategy transformation to delete rows from a virtual table.

6. In the Workflow Manager, create a session and workflow and run the workflow to extract data from the Cassandra list collection and load data to the Cassandra list.

Reading from a Cassandra Set Collection TypeYou can import a Cassandra column family as an ODBC source definition in the Designer tool and create a mapping that reads from a Cassandra set collection.

To read data from a Cassandra set collection type, perform the following tasks:

1. Import a Cassandra set source definition.

2. Create a Cassandra set mapping and run a workflow to extract data from a Cassandra set collection and load data to a flat file.

Importing a Cassandra Set Source DefinitionYou can import a Cassandra column family as an ODBC source definition in the Designer. Cassandra ODBC source definitions represent metadata for Cassandra source column families. After you import the Cassandra source definitions, you can use the source definition in a mapping that reads from Cassandra.



9



5. Click Connect.


7. Select the source table and the virtual table corresponding to the set collection type that you want to import.


The following image shows the Source Analyzer with the set source definitions imported into the Source Analyzer workspace:

Creating a Cassandra Set MappingCreate a mapping with the imported Cassandra set to read data from Cassandra and include a flat file target definition so that other business intelligence tools can consume the data.





4. Click OK.


The following image shows the virtual table set source definition, the Source Qualifier transformation, and the flat file target definition:

10


6. In the Workflow Manager, create a workflow and run the workflow to extract data from the Cassandra set collection and load data to a flat file.

Writing to a Cassandra Set Collection TypeYou can import a Cassandra column family as an ODBC target definition and create mappings in the Mapping Designer to write data to a Cassandra set collection.


To write data to a Cassandra set collection type, perform the following tasks:

1. Import a Cassandra set target definition.

2. Create a Cassandra set mapping and run a workflow to extract data from the Cassandra set and load data to a Cassandra set.

Importing a Cassandra Set Source and Target DefinitionYou can import a Cassandra column family as ODBC source and target definitions in the Designer. Cassandra ODBC source and target definitions represent metadata for Cassandra source and target column families. After you import the Cassandra target definition, you can use the target definition in a mapping that writes to Cassandra.





5. Click Connect.


7. Select the source table and the virtual table corresponding to the set collection type that you want to import.


11

The following image shows the Source Analyzer with the set source definitions imported into the Source Analyzer workspace:




12. Enter the user name and password to connect to this database.

13. Click Connect.


15. Select the target table and the virtual table corresponding to the set collection type that you want to import.


The following image shows the Target Designer with the set target definitions imported into the Target Designer workspace:

Creating a Cassandra Set MappingCreate a mapping with the imported Cassandra set source and Cassandra set target to write data to Cassandra.





4. Click OK.


For example, consider the mapping source as set_only_src_vt_col_set table and mapping target as set_only_tgt_vt_col_set table in a Cassandra set mapping

12

The following mapping shows the set write operation that you can perform:

You can perform the following write operations on a Cassandra map collection:

Insert Operation


Delete Operation


6. In the Workflow Manager, create a session and workflow and run the workflow to extract data from the Cassandra set collection and load data to the Cassandra set.

Reading from a Cassandra Map Collection TypeYou can import a Cassandra column family as an ODBC source definition in the Designer tool and create a mapping that reads from a Cassandra map collection.

To read data from a Cassandra map collection type, perform the following tasks:

1. Import a Cassandra map source definition.

2. Create a Cassandra map mapping and run a workflow to extract data from Cassandra map collection and load data to a flat file.

Importing a Cassandra Map Source DefinitionYou can import a Cassandra column family as an ODBC source definition in the Designer. Cassandra ODBC source definitions represent metadata for Cassandra source column families. After you import the Cassandra source definitions, you can use the source definition in a mapping that reads from Cassandra.





5. Click Connect.


13

7. Select the source table and the virtual table corresponding to the map collection type that you want to import.


The following image shows the Source Analyzer with the map source definitions imported into the Source Analyzer workspace:

Creating a Cassandra Map MappingCreate a mapping with the imported Cassandra map to read data from Cassandra and include a flat file target definition so that other business intelligence tools can consume the data..





4. Click OK.


The following image shows the virtual table map source definition, the Source Qualifier transformation, and the flat file target definition:


6. In the Workflow Manager, create a workflow and run the workflow to extract data from the Cassandra map collection and load data to a flat file.

14

Writing to a Cassandra Map Collection TypeYou can import a Cassandra column family as an ODBC target definition and create mappings in the Mapping Designer to write data to a Cassandra map collection.


To write data to a Cassandra list collection type, perform the following tasks:

1. Import a Cassandra map source definition.

2. Import a Cassandra map target definition.

3. Create a Cassandra map mapping and run a workflow to extract data from Cassandra map and load data to a Cassandra map.

Importing a Cassandra Map Source and Target DefinitionYou can import a Cassandra column family as ODBC source and target definitions in the Designer. Cassandra ODBC source and target definitions represent metadata for Cassandra source and target column families. After you import the Cassandra target definition, you can use the target definition in a mapping that writes to Cassandra.





5. Click Connect.


7. Select the source table and the virtual table corresponding to the map collection type that you want to import.


The following image shows the Source Analyzer with the map source definitions imported into the Source Analyzer workspace:




12. Enter the user name and password to connect to this database.

13. Click Connect.


15. Select the target table and the virtual table corresponding to the map collection type that you want to import.

15


The following image shows the Target Designer with the map target definitions imported into the Target Designer workspace:

Creating a Cassandra Map MappingCreate a mapping with the imported Cassandra list source and Cassandra list target to write data to Cassandra.





4. Click OK.


For example, consider the mapping source as map_only_src_vt_col_map table and mapping target as map_only_tgt_vt_col_map table in a Cassandra list mapping

The following mapping shows the write operation that you can perform:

You can perform the following write operations on a Cassandra map:

Insert Operation

You can use the insert operation to add one or more rows to a virtual table.


16

Update Operation

You can use a DD_UPDATE constant in an Update Strategy transformation to update rows in a virtual table.

The following image shows a data preview of the rows in the target table before you update rows in the target table:

The following image shows a data preview of the rows updated in the target virtual table:

Delete Operation


6. In the Workflow Manager, create a session and workflow and run the workflow to extract data from the Cassandra map collection and load data to the Cassandra map.

AuthorP. Keerthi MurdiyaTrainee Technical Writer

AcknowledgementsThe author would like to acknowledge Tanjot Singh Uppal, Lead QA Engineer, for his technical assistance.

17

Documents

Using PowerExchange for Cassandra to Read from and Write ... Library/1/0981...The Cassandra database supports the list, set, and map collection types. You can use PowerExchange ®