Pivotal Greenplum DatabaseContents OpenTopic 3 Contents Chapter 1: Pivotal Greenplum 5.11.3 Release Notes..... 16 Welcome to Pivotal Greenplum 5.11.3.....17 Experimental

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 5.11.3

Pivotal Greenplum DatabaseDocumentationRev: A01

© 2018 Pivotal Software, Inc.

Copyright OpenTopic

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2018 Pivotal Software, Inc. All rights reserved.

Pivotal Software, Inc. believes the information in this publication is accurate as of its publication date. Theinformation is subject to change without notice. THE INFORMATION IN THIS PUBLICATION IS PROVIDED"AS IS." PIVOTAL SOFTWARE, INC. ("Pivotal") MAKES NO REPRESENTATIONS OR WARRANTIES OF ANYKIND WITH RESPECT TO THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMSIMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any Pivotal software described in this publication requires an applicablesoftware license.

All trademarks used herein are the property of Pivotal or their respective owners.

Revised October 2018 (5.11.3)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents OpenTopic

3

Contents

Chapter 1: Pivotal Greenplum 5.11.3 Release Notes............................. 16Welcome to Pivotal Greenplum 5.11.3............................................................................................. 17Experimental Features...................................................................................................................... 18Differences Compared to Open Source Greenplum Database.........................................................19Supported Platforms..........................................................................................................................20

Veritas NetBackup.................................................................................................................. 22Supported Platform Notes...................................................................................................... 22

Pivotal Greenplum Tools and Extensions Compatibility................................................................... 24Client Tools.............................................................................................................................24Extensions...............................................................................................................................25Pivotal Greenplum Data Connectors......................................................................................26Pivotal GPText Compatibility.................................................................................................. 26Pivotal Greenplum Command Center.................................................................................... 26

Hadoop Distribution Compatibility..................................................................................................... 27Upgrading to Greenplum Database 5.11.3....................................................................................... 28

Upgrading from 5.x to 5.11.3................................................................................................. 29Troubleshooting a Failed Upgrade.........................................................................................31

Migrating Data to Pivotal Greenplum 5.x..........................................................................................32Pivotal Greenplum on DCA Systems................................................................................................34

Installing the Pivotal Greenplum 5.11.3 Software Binaries on DCA Systems........................ 34Upgrading from 5.x to 5.11.3 on DCA Systems.....................................................................34

Resolved Issues................................................................................................................................ 37Known Issues and Limitations.......................................................................................................... 38Update for gp_toolkit.gp_bloat_expected_pages Issue.................................................................... 46Update for gp_toolkit.gp_bloat_diag Issue........................................................................................48

Chapter 2: Pivotal Greenplum 5.11.2 Release Notes............................. 50Welcome to Pivotal Greenplum 5.11.2............................................................................................. 51Changed Feature...............................................................................................................................52Experimental Features...................................................................................................................... 53Differences Compared to Open Source Greenplum Database.........................................................54Supported Platforms..........................................................................................................................55




Upgrading from 5.x to 5.11.2................................................................................................. 64Troubleshooting a Failed Upgrade.........................................................................................66

Migrating Data to Pivotal Greenplum 5.x..........................................................................................67Pivotal Greenplum on DCA Systems................................................................................................69

Installing the Pivotal Greenplum 5.11.2 Software Binaries on DCA Systems........................ 69Upgrading from 5.x to 5.11.2 on DCA Systems.....................................................................69

Contents OpenTopic

4

Resolved Issues................................................................................................................................ 72Known Issues and Limitations.......................................................................................................... 73Update for gp_toolkit.gp_bloat_expected_pages Issue.................................................................... 81Update for gp_toolkit.gp_bloat_diag Issue........................................................................................83

Chapter 3: Pivotal Greenplum 5.11.1 Release Notes............................. 85Welcome to Pivotal Greenplum 5.11.1............................................................................................. 86Experimental Features...................................................................................................................... 87Differences Compared to Open Source Greenplum Database.........................................................88Supported Platforms..........................................................................................................................89




Upgrading from 5.x to 5.11.1................................................................................................. 98Troubleshooting a Failed Upgrade....................................................................................... 100

Migrating Data to Pivotal Greenplum 5.x........................................................................................101Pivotal Greenplum on DCA Systems..............................................................................................103

Installing the Pivotal Greenplum 5.11.1 Software Binaries on DCA Systems...................... 103Upgrading from 5.x to 5.11.1 on DCA Systems...................................................................103

Resolved Issues.............................................................................................................................. 106Known Issues and Limitations........................................................................................................ 107Update for gp_toolkit.gp_bloat_expected_pages Issue.................................................................. 115Update for gp_toolkit.gp_bloat_diag Issue......................................................................................117

Chapter 4: Pivotal Greenplum 5.11.0 Release Notes........................... 119Welcome to Pivotal Greenplum 5.11.0........................................................................................... 120New Features.................................................................................................................................. 121

Incremental Backup with gpbackup and gprestore.............................................................. 121Plugins for gpbackup and gprestore.................................................................................... 121GPORCA Nested Join Performance Enhancement............................................................. 122GPORCA Supports GiST Indexes........................................................................................122Enhanced Partitioned Table Statistics for GPORCA............................................................122Support for Host Names in pg_hba.conf..............................................................................122Enhancements to the Pivotal Greenplum-Kafka Connector (Experimental).........................123PXF Support for Kerberized HDFS...................................................................................... 123PXF JDBC Connector...........................................................................................................123Support for Oracle Linux 7.4 (64-bit) with RHCK.................................................................123

Changed Features...........................................................................................................................124Experimental Features.................................................................................................................... 126Deprecated Feature.........................................................................................................................127Differences Compared to Open Source Greenplum Database.......................................................128Supported Platforms........................................................................................................................129

Veritas NetBackup................................................................................................................ 131Supported Platform Notes.................................................................................................... 131

Pivotal Greenplum Tools and Extensions Compatibility................................................................. 133Client Tools...........................................................................................................................133Extensions.............................................................................................................................134

Contents OpenTopic

5

Pivotal Greenplum Data Connectors....................................................................................135Pivotal GPText Compatibility................................................................................................ 135Pivotal Greenplum Command Center.................................................................................. 135

Hadoop Distribution Compatibility................................................................................................... 136Upgrading to Greenplum Database 5.11.0..................................................................................... 137

Upgrading from 5.x to 5.11.0............................................................................................... 138Troubleshooting a Failed Upgrade....................................................................................... 140

Migrating Data to Pivotal Greenplum 5.x........................................................................................141Pivotal Greenplum on DCA Systems..............................................................................................143

Installing the Pivotal Greenplum 5.11.0 Software Binaries on DCA Systems...................... 143Upgrading from 5.x to 5.11.0 on DCA Systems...................................................................143

Resolved Issues.............................................................................................................................. 146Known Issues and Limitations........................................................................................................ 151Update for gp_toolkit.gp_bloat_expected_pages Issue.................................................................. 159Update for gp_toolkit.gp_bloat_diag Issue......................................................................................161

Chapter 5: Greenplum Database Installation Guide.............................163Introduction to Greenplum...............................................................................................................164

The Greenplum Master.........................................................................................................164The Segments...................................................................................................................... 165The Interconnect...................................................................................................................166ETL Hosts for Data Loading................................................................................................ 167Greenplum Performance Monitoring.................................................................................... 168

Estimating Storage Capacity........................................................................................................... 169Calculating Usable Disk Capacity........................................................................................ 169Calculating User Data Size.................................................................................................. 169Calculating Space Requirements for Metadata and Logs....................................................170

Configuring Your Systems and Installing Greenplum..................................................................... 171System Requirements...........................................................................................................171Setting the Greenplum Recommended OS Parameters...................................................... 173Creating the Greenplum Database Administrative User Account........................................ 177Installing the Greenplum Database Software.......................................................................178Installing and Configuring Greenplum on all Hosts..............................................................179Installing Oracle Compatibility Functions............................................................................. 181Installing Optional Modules.................................................................................................. 181Installing Greenplum Database Extensions..........................................................................182Installing and Configuring the Greenplum Platform Extension Framework (PXF)................ 182Creating the Data Storage Areas.........................................................................................183Synchronizing System Clocks.............................................................................................. 184Enabling iptables.................................................................................................................. 184Amazon EC2 Configuration (Amazon Web Services)..........................................................187Next Steps............................................................................................................................ 192

Installing the Data Science Packages............................................................................................ 193Python Data Science Module Package................................................................................193R Data Science Library Package......................................................................................... 195

Validating Your Systems................................................................................................................. 198Validating OS Settings..........................................................................................................198Validating Hardware Performance........................................................................................198Validating Disk I/O and Memory Bandwidth.........................................................................199

Initializing a Greenplum Database System..................................................................................... 201Overview............................................................................................................................... 201Initializing Greenplum Database...........................................................................................201Setting Greenplum Environment Variables.......................................................................... 204Next Steps............................................................................................................................ 205

Configuring Timezone and Localization Settings............................................................................207

Contents OpenTopic

6

Configuring the Timezone.................................................................................................... 207About Locale Support in Greenplum Database................................................................... 207Character Set Support..........................................................................................................209Setting the Character Set.....................................................................................................211Character Set Conversion Between Server and Client........................................................212

About Implicit Text Casting in Greenplum Database......................................................................215Workaround: Manually Creating Missing Operators.............................................................216

Installation Management Utilities.....................................................................................................219Greenplum Environment Variables................................................................................................. 220

Required Environment Variables..........................................................................................220Optional Environment Variables........................................................................................... 220

Chapter 6: Greenplum Database Administrator Guide........................ 222Greenplum Database Concepts...................................................................................................... 223

About the Greenplum Architecture....................................................................................... 223About Management and Monitoring Utilities........................................................................ 225About Concurrency Control in Greenplum Database...........................................................226About Parallel Data Loading................................................................................................ 234About Redundancy and Failover in Greenplum Database...................................................235About Database Statistics in Greenplum Database............................................................. 237

Managing a Greenplum System..................................................................................................... 244About the Greenplum Database Release Version Number................................................. 244Starting and Stopping Greenplum Database....................................................................... 244Accessing the Database.......................................................................................................247Configuring the Greenplum Database System.....................................................................255Enabling High Availability and Data Consistency Features................................................. 266Backing Up and Restoring Databases................................................................................. 284Expanding a Greenplum System..........................................................................................343Migrating Data...................................................................................................................... 358Monitoring a Greenplum System..........................................................................................369Routine System Maintenance Tasks....................................................................................389Recommended Monitoring and Maintenance Tasks............................................................ 393

Managing Greenplum Database Access.........................................................................................401Configuring Client Authentication......................................................................................... 401Managing Roles and Privileges............................................................................................429

Defining Database Objects..............................................................................................................436Creating and Managing Databases......................................................................................436Creating and Managing Tablespaces...................................................................................438Creating and Managing Schemas........................................................................................441Creating and Managing Tables............................................................................................ 442Choosing the Table Storage Model..................................................................................... 445Partitioning Large Tables......................................................................................................456Creating and Using Sequences........................................................................................... 468Using Indexes in Greenplum Database............................................................................... 471Creating and Managing Views............................................................................................. 474

Distribution and Skew..................................................................................................................... 475Local (Co-located) Joins.......................................................................................................475Data Skew............................................................................................................................ 475Processing Skew.................................................................................................................. 476

Inserting, Updating, and Deleting Data...........................................................................................479About Concurrency Control in Greenplum Database...........................................................479Inserting Rows...................................................................................................................... 480Updating Existing Rows........................................................................................................481Deleting Rows.......................................................................................................................481Working With Transactions...................................................................................................481

Contents OpenTopic

7

Vacuuming the Database..................................................................................................... 483Querying Data................................................................................................................................. 484

About Greenplum Query Processing....................................................................................484About GPORCA....................................................................................................................487Defining Queries................................................................................................................... 500WITH Queries (Common Table Expressions)......................................................................510Using Functions and Operators............................................................................................512Working with JSON Data..................................................................................................... 522Working with XML Data........................................................................................................526Query Performance.............................................................................................................. 538Managing Spill Files Generated by Queries........................................................................ 538Query Profiling...................................................................................................................... 538

Working with External Data.............................................................................................................544Defining External Tables...................................................................................................... 544Accessing External Data with PXF...................................................................................... 562Accessing HDFS Data with gphdfs...................................................................................... 563Using the Greenplum Parallel File Server (gpfdist)..............................................................586

Loading and Unloading Data.......................................................................................................... 590Loading Data Using an External Table................................................................................ 591Loading and Writing Non-HDFS Custom Data.................................................................... 591Handling Load Errors............................................................................................................594Loading Data with gpload.....................................................................................................596Transforming External Data with gpfdist and gpload........................................................... 597Loading Data with COPY..................................................................................................... 607Running COPY in Single Row Error Isolation Mode............................................................607Optimizing Data Load and Query Performance................................................................... 607Unloading Data from Greenplum Database......................................................................... 608Formatting Data Files........................................................................................................... 610Example Custom Data Access Protocol.............................................................................. 613

Managing Performance................................................................................................................... 620Defining Database Performance.......................................................................................... 620Common Causes of Performance Issues............................................................................ 621Greenplum Database Memory Overview............................................................................. 624Managing Resources............................................................................................................627Investigating a Performance Problem.................................................................................. 655

Chapter 7: Greenplum Database Security Configuration Guide......... 659Securing the Database....................................................................................................................660Greenplum Database Ports and Protocols..................................................................................... 661Configuring Client Authentication.................................................................................................... 665

Allowing Connections to Greenplum Database....................................................................665Editing the pg_hba.conf File.................................................................................................667Authentication Methods........................................................................................................ 667SSL Client Authentication.....................................................................................................670PAM Based Authentication...................................................................................................672Radius Authentication...........................................................................................................672Limiting Concurrent Connections......................................................................................... 673Encrypting Client/Server Connections..................................................................................674

Configuring Database Authorization................................................................................................675Access Permissions and Roles............................................................................................675Managing Object Privileges..................................................................................................675Using SSH-256 Encryption...................................................................................................676Restricting Access by Time..................................................................................................678Dropping a Time-based Restriction.................................................................................... 680

Greenplum Command Center Security........................................................................................... 681

Contents OpenTopic

8

Auditing............................................................................................................................................ 684Encrypting Data and Database Connections.................................................................................. 689

Encrypting gpfdist Connections............................................................................................ 689Encrypting Data at Rest with pgcrypto.................................................................................690

Enabling gphdfs Authentication with a Kerberos-secured Hadoop Cluster.....................................698Prerequisites......................................................................................................................... 698Configuring the Greenplum Cluster......................................................................................698Creating and Installing Keytab Files.................................................................................... 699Configuring gphdfs for Kerberos.......................................................................................... 700Testing Greenplum Database Access to HDFS...................................................................701Troubleshooting HDFS with Kerberos..................................................................................702

Security Best Practices................................................................................................................... 704

Chapter 8: Greenplum Database Best Practices..................................708Best Practices Summary.................................................................................................................709System Configuration...................................................................................................................... 715Schema Design............................................................................................................................... 720

Data Types........................................................................................................................... 720Storage Model...................................................................................................................... 720Compression......................................................................................................................... 721Distributions.......................................................................................................................... 722Partitioning............................................................................................................................ 725Indexes..................................................................................................................................727Column Sequence and Byte Alignment............................................................................... 727

Memory and Resource Management with Resource Queues........................................................ 729Memory and Resource Management with Resource Groups......................................................... 733System Monitoring and Maintenance..............................................................................................736

Monitoring............................................................................................................................. 736Updating Statistics with ANALYZE.......................................................................................737Managing Bloat in the Database..........................................................................................738Monitoring Greenplum Database Log Files..........................................................................742

Loading Data................................................................................................................................... 744INSERT Statement with Column Values..............................................................................744COPY Statement.................................................................................................................. 744External Tables.....................................................................................................................744External Tables with Gpfdist................................................................................................ 744Gpload...................................................................................................................................745Best Practices.......................................................................................................................746

Migrating Data with gptransfer........................................................................................................ 747Security............................................................................................................................................ 753Encrypting Data and Database Connections.................................................................................. 756Tuning SQL Queries....................................................................................................................... 765

How to Generate Explain Plans........................................................................................... 765How to Read Explain Plans................................................................................................. 765Optimizing Greenplum Queries............................................................................................ 767

High Availability............................................................................................................................... 769Disk Storage......................................................................................................................... 769Master Mirroring....................................................................................................................769Segment Mirroring................................................................................................................ 770Dual Clusters........................................................................................................................ 771Backup and Restore.............................................................................................................771Detecting Failed Master and Segment Instances................................................................ 772Segment Mirroring Configuration..........................................................................................773

Contents OpenTopic

9

Chapter 9: Greenplum Database Utility Guide..................................... 779Management Utility Reference........................................................................................................ 780

Backend Server Programs................................................................................................... 781analyzedb..............................................................................................................................782gpactivatestandby................................................................................................................. 786gpaddmirrors......................................................................................................................... 788gpbackup...............................................................................................................................791gpcheck.................................................................................................................................796gpcheckcat............................................................................................................................ 798gpcheckperf...........................................................................................................................801gpconfig.................................................................................................................................804gpcrondump.......................................................................................................................... 807gpdbrestore........................................................................................................................... 821gpcopy...................................................................................................................................828gpdeletesystem..................................................................................................................... 837gpexpand.............................................................................................................................. 838gpfdist....................................................................................................................................841gpfilespace............................................................................................................................ 844gpinitstandby......................................................................................................................... 847gpinitsystem.......................................................................................................................... 849gpload................................................................................................................................... 857gplogfilter...............................................................................................................................867gpmapreduce........................................................................................................................ 870gpmfr..................................................................................................................................... 871gpmovemirrors...................................................................................................................... 874gpperfmon_install..................................................................................................................876gppkg.................................................................................................................................... 880gprecoverseg........................................................................................................................ 882gpreload................................................................................................................................ 886gprestore............................................................................................................................... 888gpscp.....................................................................................................................................893gpseginstall........................................................................................................................... 895gpssh.....................................................................................................................................897gpssh-exkeys........................................................................................................................ 900gpstart................................................................................................................................... 902gpstate.................................................................................................................................. 904gpstop................................................................................................................................... 908gpsys1...................................................................................................................................910gptransfer.............................................................................................................................. 911pgbouncer............................................................................................................................. 923pgbouncer.ini.........................................................................................................................924pgbouncer-admin.................................................................................................................. 937

Client Utility Reference....................................................................................................................946Client Utility Summary.......................................................................................................... 946

Additional Supplied Modules........................................................................................................... 996citext Data Type................................................................................................................... 996dblink Functions....................................................................................................................998hstore Functions................................................................................................................... 999Oracle Compatibility Functions...........................................................................................1002passwordcheck................................................................................................................... 1023

Chapter 10: Greenplum Database Reference Guide.......................... 1025SQL Command Reference............................................................................................................ 1026

Contents OpenTopic

10

SQL Syntax Summary........................................................................................................1028ABORT................................................................................................................................1055ALTER AGGREGATE.........................................................................................................1056ALTER CONVERSION....................................................................................................... 1057ALTER DATABASE............................................................................................................ 1058ALTER DOMAIN.................................................................................................................1059ALTER EXTENSION...........................................................................................................1061ALTER EXTERNAL TABLE................................................................................................1064ALTER FILESPACE............................................................................................................1065ALTER FUNCTION.............................................................................................................1066ALTER GROUP.................................................................................................................. 1068ALTER INDEX.................................................................................................................... 1069ALTER LANGUAGE........................................................................................................... 1071ALTER OPERATOR........................................................................................................... 1071ALTER OPERATOR CLASS.............................................................................................. 1072ALTER OPERATOR FAMILY.............................................................................................1073ALTER PROTOCOL........................................................................................................... 1075ALTER RESOURCE GROUP............................................................................................ 1076ALTER RESOURCE QUEUE.............................................................................................1079ALTER ROLE..................................................................................................................... 1081ALTER SCHEMA................................................................................................................1085ALTER SEQUENCE........................................................................................................... 1085ALTER TABLE....................................................................................................................1087ALTER TABLESPACE........................................................................................................1097ALTER TYPE......................................................................................................................1098ALTER USER..................................................................................................................... 1099ALTER VIEW...................................................................................................................... 1100ANALYZE............................................................................................................................1101BEGIN................................................................................................................................. 1104CHECKPOINT.....................................................................................................................1106CLOSE................................................................................................................................ 1106CLUSTER........................................................................................................................... 1107COMMENT..........................................................................................................................1109COMMIT..............................................................................................................................1111COPY.................................................................................................................................. 1111CREATE AGGREGATE......................................................................................................1122CREATE CAST...................................................................................................................1126CREATE CONVERSION.................................................................................................... 1128CREATE DATABASE......................................................................................................... 1129CREATE DOMAIN..............................................................................................................1131CREATE EXTENSION........................................................................................................1132CREATE EXTERNAL TABLE.............................................................................................1134CREATE FUNCTION..........................................................................................................1142CREATE GROUP............................................................................................................... 1148CREATE INDEX................................................................................................................. 1149CREATE LANGUAGE........................................................................................................ 1152CREATE OPERATOR........................................................................................................ 1154CREATE OPERATOR CLASS........................................................................................... 1158CREATE OPERATOR FAMILY..........................................................................................1162CREATE PROTOCOL........................................................................................................ 1163CREATE RESOURCE GROUP......................................................................................... 1164CREATE RESOURCE QUEUE..........................................................................................1167CREATE ROLE.................................................................................................................. 1170CREATE RULE...................................................................................................................1174CREATE SCHEMA.............................................................................................................1176CREATE SEQUENCE........................................................................................................ 1177

Contents OpenTopic

11

CREATE TABLE.................................................................................................................1180CREATE TABLE AS...........................................................................................................1192CREATE TABLESPACE.....................................................................................................1195CREATE TYPE...................................................................................................................1196CREATE USER.................................................................................................................. 1201CREATE VIEW................................................................................................................... 1202DEALLOCATE.................................................................................................................... 1204DECLARE........................................................................................................................... 1204DELETE.............................................................................................................................. 1207DISCARD............................................................................................................................ 1209DO.......................................................................................................................................1210DROP AGGREGATE..........................................................................................................1211DROP CAST.......................................................................................................................1212DROP CONVERSION........................................................................................................ 1213DROP DATABASE............................................................................................................. 1213DROP DOMAIN.................................................................................................................. 1214DROP EXTENSION............................................................................................................1215DROP EXTERNAL TABLE.................................................................................................1216DROP FILESPACE.............................................................................................................1217DROP FUNCTION..............................................................................................................1217DROP GROUP................................................................................................................... 1218DROP INDEX..................................................................................................................... 1219DROP LANGUAGE.............................................................................................................1219DROP OPERATOR............................................................................................................ 1220DROP OPERATOR CLASS............................................................................................... 1221DROP OPERATOR FAMILY.............................................................................................. 1222DROP OWNED...................................................................................................................1223DROP PROTOCOL............................................................................................................ 1224DROP RESOURCE GROUP..............................................................................................1224DROP RESOURCE QUEUE.............................................................................................. 1225DROP ROLE.......................................................................................................................1226DROP RULE.......................................................................................................................1227DROP SCHEMA................................................................................................................. 1228DROP SEQUENCE............................................................................................................ 1229DROP TABLE..................................................................................................................... 1229DROP TABLESPACE.........................................................................................................1230DROP TYPE....................................................................................................................... 1231DROP USER...................................................................................................................... 1232DROP VIEW....................................................................................................................... 1232END.....................................................................................................................................1233EXECUTE........................................................................................................................... 1233EXPLAIN............................................................................................................................. 1234FETCH................................................................................................................................ 1237GRANT................................................................................................................................1239INSERT............................................................................................................................... 1243LOAD.................................................................................................................................. 1245LOCK.................................................................................................................................. 1246MOVE..................................................................................................................................1249PREPARE........................................................................................................................... 1250REASSIGN OWNED...........................................................................................................1252REINDEX............................................................................................................................ 1252RELEASE SAVEPOINT......................................................................................................1254RESET................................................................................................................................ 1255REVOKE............................................................................................................................. 1255ROLLBACK......................................................................................................................... 1257ROLLBACK TO SAVEPOINT.............................................................................................1258

Contents OpenTopic

12

SAVEPOINT........................................................................................................................1259SELECT.............................................................................................................................. 1260SELECT INTO.................................................................................................................... 1275SET..................................................................................................................................... 1276SET ROLE.......................................................................................................................... 1278SET SESSION AUTHORIZATION..................................................................................... 1279SET TRANSACTION.......................................................................................................... 1280SHOW................................................................................................................................. 1282START TRANSACTION..................................................................................................... 1283TRUNCATE.........................................................................................................................1284UPDATE..............................................................................................................................1285VACUUM.............................................................................................................................1288VALUES.............................................................................................................................. 1291

SQL 2008 Optional Feature Compliance......................................................................................1293Greenplum Environment Variables............................................................................................... 1322

Required Environment Variables........................................................................................1322Optional Environment Variables......................................................................................... 1322

System Catalog Reference........................................................................................................... 1324System Tables.................................................................................................................... 1324System Views..................................................................................................................... 1325System Catalogs Definitions...............................................................................................1326

The gp_toolkit Administrative Schema..........................................................................................1411Checking for Tables that Need Routine Maintenance........................................................1411Checking for Locks.............................................................................................................1412Checking Append-Optimized Tables.................................................................................. 1414Viewing Greenplum Database Server Log Files................................................................ 1418Checking Server Configuration Files..................................................................................1421Checking for Failed Segments........................................................................................... 1422Checking Resource Group Activity and Status.................................................................. 1423Checking Resource Queue Activity and Status................................................................. 1425Checking Query Disk Spill Space Usage...........................................................................1427Viewing Users and Groups (Roles)....................................................................................1429Checking Database Object Sizes and Disk Space............................................................ 1430Checking for Uneven Data Distribution.............................................................................. 1434

The gpperfmon Database..............................................................................................................1435database_*.........................................................................................................................1437diskspace_*....................................................................................................................... 1438interface_stats_*................................................................................................................ 1438log_alert_*..........................................................................................................................1440queries_*............................................................................................................................. 1441segment_*..........................................................................................................................1443socket_stats_*.....................................................................................................................1444system_*............................................................................................................................. 1445dynamic_memory_info........................................................................................................ 1447memory_info...................................................................................................................... 1447

Greenplum Database Data Types.................................................................................................1449Character Set Support...................................................................................................................1454

Setting the Character Set...................................................................................................1456Character Set Conversion Between Server and Client...................................................... 1456

Server Configuration Parameters..................................................................................................1459Parameter Types and Values.............................................................................................1459Setting Parameters............................................................................................................. 1459Parameter Categories.........................................................................................................1460Configuration Parameters...................................................................................................1470

Summary of Built-in Functions...................................................................................................... 1552Greenplum Database Function Types................................................................................1552

Contents OpenTopic

13

Built-in Functions and Operators........................................................................................1553JSON Functions and Operators......................................................................................... 1556Window Functions.............................................................................................................. 1559Advanced Aggregate Functions......................................................................................... 1561

Greenplum MapReduce Specification...........................................................................................1563Greenplum MapReduce Document Format........................................................................1563Greenplum MapReduce Document Schema......................................................................1564Example Greenplum MapReduce Document..................................................................... 1571

Greenplum PL/pgSQL Procedural Language............................................................................... 1577About Greenplum Database PL/pgSQL............................................................................. 1577PL/pgSQL Plan Caching.....................................................................................................1579PL/pgSQL Examples...........................................................................................................1579References..........................................................................................................................1583

Greenplum PostGIS Extension..................................................................................................... 1584About PostGIS.................................................................................................................... 1584Enabling and Removing PostGIS Support......................................................................... 1585Usage..................................................................................................................................1586PostGIS Extension Support and Limitations...................................................................... 1587PostGIS Support Scripts.....................................................................................................1588

Greenplum PL/R Language Extension..........................................................................................1591About Greenplum Database PL/R......................................................................................1591

Greenplum PL/Python Language Extension................................................................................. 1597About Greenplum PL/Python..............................................................................................1597Enabling and Removing PL/Python support...................................................................... 1597Developing Functions with PL/Python................................................................................1598Installing Python Modules...................................................................................................1601Examples............................................................................................................................ 1604References..........................................................................................................................1606

Greenplum PL/Container Language Extension.............................................................................1607About the PL/Container Language Extension.................................................................... 1607About PL/Container Resource Management......................................................................1608PL/Container Docker Images............................................................................................. 1610Prerequisites....................................................................................................................... 1610Installing the PL/Container Language Extension............................................................... 1611Installing PL/Container Docker Images.............................................................................. 1614Uninstalling PL/Container................................................................................................... 1614Using PL/Container Functions............................................................................................1616About PL/Container Running PL/Python............................................................................ 1618About PL/Container Running PL/R.....................................................................................1619Configuring PL/Container....................................................................................................1619Installing Docker................................................................................................................. 1629References..........................................................................................................................1630

Greenplum PL/Java Language Extension.....................................................................................1631About PL/Java.................................................................................................................... 1631About Greenplum Database PL/Java.................................................................................1632Installing PL/Java................................................................................................................1633Uninstalling PL/Java........................................................................................................... 1634Enabling PL/Java and Installing JAR Files........................................................................ 1635Writing PL/Java functions................................................................................................... 1635Using JDBC........................................................................................................................ 1641Exception Handling.............................................................................................................1641Savepoints.......................................................................................................................... 1641Logging............................................................................................................................... 1642Security............................................................................................................................... 1642Some PL/Java Issues and Solutions..................................................................................1643Example.............................................................................................................................. 1644

Contents OpenTopic

14

References..........................................................................................................................1645Greenplum PL/Perl Language Extension......................................................................................1646

About Greenplum PL/Perl...................................................................................................1646Greenplum Database PL/Perl Limitations.......................................................................... 1646Trusted/Untrusted Language.............................................................................................. 1646Enabling and Removing PL/Perl Support...........................................................................1647Developing Functions with PL/Perl.....................................................................................1647

Greenplum MADlib Extension for Analytics.................................................................................. 1651About MADlib......................................................................................................................1651Installing MADlib................................................................................................................. 1651Upgrading MADlib...............................................................................................................1652Uninstalling MADlib.............................................................................................................1653Examples............................................................................................................................ 1653References..........................................................................................................................1660

Greenplum Partner Connector API............................................................................................... 1661Using the GPPC API..........................................................................................................1661Building a GPPC Shared Library with PGXS.....................................................................1673Registering a GPPC Function with Greenplum Database................................................. 1673Packaging and Deployment Considerations.......................................................................1674GPPC Text Function Example........................................................................................... 1675GPPC Set-Returning Function Example............................................................................ 1677

Greenplum Fuzzy String Match Extension....................................................................................1681Soundex Functions............................................................................................................. 1681Levenshtein Functions........................................................................................................1682Metaphone Functions......................................................................................................... 1682Double Metaphone Functions.............................................................................................1683Installing and Uninstalling the Fuzzy String Match Functions............................................ 1683

Summary of Greenplum Features.................................................................................................1684Greenplum SQL Standard Conformance........................................................................... 1684Greenplum and PostgreSQL Compatibility.........................................................................1686

Chapter 11: Greenplum Database UNIX Client Documentation........ 1695Greenplum Database Client Tools for UNIX.................................................................................1696

Installing the Greenplum Client Tools................................................................................ 1696Client Tools Reference.......................................................................................................1699

Greenplum Database Load Tools for UNIX..................................................................................1700Installing the Greenplum Load Tools................................................................................. 1700Load Tools Reference........................................................................................................ 1701

Chapter 12: Greenplum Database Windows Client Documentation..1703Greenplum Database Client Tools for Windows...........................................................................1704

Installing the Greenplum Client Tools................................................................................ 1704Running the Greenplum Client Tools.................................................................................1707Client Tools Reference.......................................................................................................1708

Greenplum Database Load Tools for Windows............................................................................ 1710Installing Greenplum Loader.............................................................................................. 1710Running Greenplum Loader............................................................................................... 1712Running gpfdist as a Windows Service..............................................................................1716Loader Program Reference................................................................................................ 1717

Chapter 13: DataDirect ODBC Drivers for Pivotal Greenplum...........1718Prerequisites.................................................................................................................................. 1719Supported Client Platforms........................................................................................................... 1720

Contents OpenTopic

15

Installing on Linux Systems.......................................................................................................... 1721Configuring the Driver on Linux......................................................................................... 1722Testing the Driver Connection on Linux.............................................................................1723

Installing on Windows Systems.................................................................................................... 1724Verifying the Version on Windows..................................................................................... 1724Configuring and Testing the Driver on Windows................................................................1724

DataDirect Driver Documentation..................................................................................................1726

Chapter 14: DataDirect JDBC Driver for Pivotal Greenplum............. 1727Prerequisites.................................................................................................................................. 1728Downloading the DataDirect JDBC Driver.................................................................................... 1729Obtaining Version Details for the Driver....................................................................................... 1730Usage Information......................................................................................................................... 1731DataDirect Driver Documentation..................................................................................................1732

Pivotal Greenplum 5.11.3 Release Notes OpenTopic

16

Chapter 1

Pivotal Greenplum 5.11.3 Release Notes

Updated: October, 2018

• Welcome to Pivotal Greenplum 5.11.3• Experimental Features• Differences Compared to Open Source Greenplum Database• Supported Platforms• Pivotal Greenplum Tools and Extensions Compatibility• Hadoop Distribution Compatibility• Upgrading to Greenplum Database 5.11.3• Migrating Data to Pivotal Greenplum 5.x• Pivotal Greenplum on DCA Systems• Resolved Issues• Known Issues and Limitations• Update for gp_toolkit.gp_bloat_expected_pages Issue• Update for gp_toolkit.gp_bloat_diag Issue


17

Welcome to Pivotal Greenplum 5.11.3Pivotal Greenplum Database is a massively parallel processing (MPP) database server that supports nextgeneration data warehousing and large-scale analytics processing. By automatically partitioning dataand running parallel queries, it allows a cluster of servers to operate as a single database supercomputerperforming tens or hundreds times faster than a traditional database. It supports SQL, MapReduce parallelprocessing, and data volumes ranging from hundreds of gigabytes, to hundreds of terabytes.

This document contains pertinent release information about Pivotal Greenplum Database 5.11.3. Forprevious versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see the Pivotal Support LifecyclePolicy.

Pivotal Greenplum 5.x software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 5.x is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

Pivotal Greenplum 5.11.3 is a patch release that includes a resolved issue.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://pivotal.io/support/lifecycle_policyhttps://pivotal.io/support/lifecycle_policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


18

Experimental FeaturesBecause Pivotal Greenplum Database is based on the open source Greenplum Database project code,it includes several experimental features to allow interested developers to experiment with their use ondevelopment systems. Feedback will help drive development of these features, and they may becomesupported in future versions of the product.

Warning: Experimental features are not recommended or supported for production deployments.These features may change in or be removed from future versions of the product based on furthertesting and feedback. Moreover, any features that may be visible in the open source code butthat are not described in the product documentation should be considered experimental andunsupported for production use.

Greenplum Database 5.11.3 includes these experimental features:

• Storage plugin API for gpbackup and gprestore. Partners, customers, and OSS developers candevelop plugins to use in conjunction with gpbackup and gprestore.

For information about the storage plugin API, see Backup/Restore Storage Plugin API in the PivotalGreenplum Database Documentation.

• Recursive WITH Queries (Common Table Expressions). See WITH Queries (Common TableExpressions) in the Pivotal Greenplum Database Documentation.

• Resource groups remain an experimental feature only on the SuSE 11 platform, due to limited cgroupsfunctionality in the kernel.

SuSE 12 resolves the Linux cgroup issues that caused the performance degradation when GreenplumDatabase resource groups are enabled.

• Integration with the Pivotal Greenplum-Kafka Connector (experimental). The Connector provideshigh speed, parallel data transfer from a Kafka cluster to a Pivotal Greenplum Database cluster forbatch and streaming ETL operations. Refer to the Pivotal Greenplum-Kafka Connector (Experimental)documentation for more information about this feature.

http://greenplum.org/https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/../greenplum-kafka/intro.html


19

Differences Compared to Open Source GreenplumDatabase

Pivotal Greenplum 5.x includes all of the functionality in the open source Greenplum Database project andadds:

• Product packaging and install

Documents

Pivotal Greenplum DatabaseContents OpenTopic 3 Contents Chapter 1: Pivotal Greenplum 5.11.3 Release Notes..... 16 Welcome to Pivotal Greenplum 5.11.3.....17 Experimental