Pivotal Greenplum Database · Contents Release Notes 4 Initializing a Greenplum Database System.....62

PRODUCT DOCUMENTATION

Pivotal™ GreenplumDatabase®Version 6.8

Pivotal Greenplum DatabaseDocumentationRev: A02

© 2020 Pivotal Software, Inc.

Copyright Release Notes

2

Notice

Copyright

Privacy Policy | Terms of Use

Copyright © 2020 VMware, Inc. or its affiliates. All rights reserved.

Revised June 2020 (6.8.1)

http://pivotal.io/privacy-policyhttp://pivotal.io/terms-of-use

Contents Release Notes

3

Contents

Chapter 2: Pivotal Greenplum 6.8 Release Notes.................................. 14Release 6.8.1.................................................................................................................................... 15

Resolved Issues..................................................................................................................... 15Release 6.8.0.................................................................................................................................... 16

Features.................................................................................................................................. 16Resolved Issues..................................................................................................................... 17Upgrading from Greenplum 6.x to Greenplum 6.8.................................................................18

Deprecated Features.........................................................................................................................19Migrating Data to Greenplum 6........................................................................................................ 21Known Issues and Limitations.......................................................................................................... 22Differences Compared to Open Source Greenplum Database.........................................................24

Chapter 3: Installing and Upgrading Greenplum................................... 25Platform Requirements......................................................................................................................26

Operating Systems................................................................................................................. 26Hardware and Network...........................................................................................................28Storage....................................................................................................................................28Tools and Extensions Compatibility....................................................................................... 29Hadoop Distributions.............................................................................................................. 31

Introduction to Greenplum.................................................................................................................32The Greenplum Master...........................................................................................................33The Segments........................................................................................................................ 33The Interconnect.....................................................................................................................37ETL Hosts for Data Loading.................................................................................................. 39Greenplum Performance Monitoring...................................................................................... 40

Estimating Storage Capacity............................................................................................................. 42Calculating Usable Disk Capacity.......................................................................................... 42Calculating User Data Size.................................................................................................... 43Calculating Space Requirements for Metadata and Logs......................................................43

Configuring Your Systems.................................................................................................................44Disabling SELinux and Firewall Software.............................................................................. 44Recommended OS Parameters Settings............................................................................... 45Synchronizing System Clocks................................................................................................ 52Creating the Greenplum Administrative User.........................................................................53Next Steps.............................................................................................................................. 54

Installing the Greenplum Database Software................................................................................... 55Installing Greenplum Database.............................................................................................. 55Enabling Passwordless SSH.................................................................................................. 56Confirming Your Installation................................................................................................... 57About Your Greenplum Database Installation........................................................................57Next Steps.............................................................................................................................. 57

Creating the Data Storage Areas......................................................................................................58Creating Data Storage Areas on the Master and Standby Master Hosts...............................58Creating Data Storage Areas on Segment Hosts.................................................................. 58Next Steps.............................................................................................................................. 59

Validating Your Systems................................................................................................................... 60Validating Network Performance............................................................................................ 60Validating Disk I/O and Memory Bandwidth...........................................................................61


4

Initializing a Greenplum Database System.......................................................................................62Overview................................................................................................................................. 62Initializing Greenplum Database.............................................................................................62Setting Greenplum Environment Variables............................................................................ 66Next Steps.............................................................................................................................. 67

Installing Optional Extensions........................................................................................................... 68Procedural Language, Machine Learning, and Geospatial Extensions..................................68Python Data Science Module Package..................................................................................68R Data Science Library Package........................................................................................... 72Greenplum Platform Extension Framework (PXF)................................................................. 75

Installing Additional Supplied Modules..............................................................................................76Configuring Timezone and Localization Settings..............................................................................77

Configuring the Timezone...................................................................................................... 77About Locale Support in Greenplum Database..................................................................... 77Character Set Support............................................................................................................79Setting the Character Set.......................................................................................................81Character Set Conversion Between Server and Client..........................................................82

Upgrading to Greenplum 6................................................................................................................85Upgrading from an Earlier Greenplum 6 Release..................................................................85Migrating Data from Greenplum 4.3 or 5 to Greenplum 6..................................................... 87

Enabling iptables (Optional).............................................................................................................. 94Example iptables Rules..........................................................................................................94

Installation Management Utilities.......................................................................................................97Greenplum Environment Variables................................................................................................... 98

Required Environment Variables............................................................................................98Optional Environment Variables............................................................................................. 98

Example Ansible Playbook..............................................................................................................100

Chapter 4: Greenplum Database Administrator Guide........................ 102Greenplum Database Concepts...................................................................................................... 103

About the Greenplum Architecture....................................................................................... 103About Management and Monitoring Utilities........................................................................ 105About Concurrency Control in Greenplum Database...........................................................106About Parallel Data Loading................................................................................................ 114About Redundancy and Failover in Greenplum Database...................................................115About Database Statistics in Greenplum Database............................................................. 117

Managing a Greenplum System..................................................................................................... 125About the Greenplum Database Release Version Number................................................. 125Starting and Stopping Greenplum Database....................................................................... 125Accessing the Database.......................................................................................................128Configuring the Greenplum Database System.....................................................................136Enabling Compression..........................................................................................................138Enabling High Availability and Data Consistency Features................................................. 139Backing Up and Restoring Databases................................................................................. 156Expanding a Greenplum System..........................................................................................196Migrating Data with gpcopy..................................................................................................211Monitoring a Greenplum System..........................................................................................211Routine System Maintenance Tasks....................................................................................227Recommended Monitoring and Maintenance Tasks............................................................ 231

Managing Greenplum Database Access.........................................................................................238Configuring Client Authentication......................................................................................... 238Managing Roles and Privileges............................................................................................257

Defining Database Objects..............................................................................................................264Creating and Managing Databases......................................................................................264Creating and Managing Tablespaces...................................................................................266


5

Creating and Managing Schemas........................................................................................268Creating and Managing Tables............................................................................................ 270Choosing the Table Storage Model..................................................................................... 275Partitioning Large Tables......................................................................................................286Creating and Using Sequences........................................................................................... 299Using Indexes in Greenplum Database............................................................................... 302Creating and Managing Views............................................................................................. 306Creating and Managing Materialized Views.........................................................................313

Distribution and Skew..................................................................................................................... 315Local (Co-located) Joins.......................................................................................................315Data Skew............................................................................................................................ 315Processing Skew.................................................................................................................. 316

Inserting, Updating, and Deleting Data...........................................................................................319About Concurrency Control in Greenplum Database...........................................................319Inserting Rows...................................................................................................................... 320Updating Existing Rows........................................................................................................321Deleting Rows.......................................................................................................................321Working With Transactions...................................................................................................321Global Deadlock Detector.....................................................................................................323Vacuuming the Database..................................................................................................... 325Running Out of Locks...........................................................................................................325

Querying Data................................................................................................................................. 327About Greenplum Query Processing....................................................................................327About GPORCA....................................................................................................................330Defining Queries................................................................................................................... 343WITH Queries (Common Table Expressions)......................................................................355Using Functions and Operators............................................................................................359Working with JSON Data..................................................................................................... 370Working with XML Data........................................................................................................383Using Full Text Search.........................................................................................................395Using Greenplum MapReduce............................................................................................. 430Query Performance.............................................................................................................. 438Managing Spill Files Generated by Queries........................................................................ 438Query Profiling...................................................................................................................... 438

Working with External Data.............................................................................................................444Accessing External Data with PXF...................................................................................... 444Defining External Tables...................................................................................................... 444Accessing External Data with Foreign Tables..................................................................... 462Using the Greenplum Parallel File Server (gpfdist)..............................................................471

Loading and Unloading Data.......................................................................................................... 475Loading Data Using an External Table................................................................................ 476Loading and Writing Non-HDFS Custom Data.................................................................... 476Handling Load Errors............................................................................................................479Loading Data with gpload.....................................................................................................481Accessing External Data with PXF...................................................................................... 482Transforming External Data with gpfdist and gpload........................................................... 483Loading Data with COPY..................................................................................................... 493Running COPY in Single Row Error Isolation Mode............................................................494Optimizing Data Load and Query Performance................................................................... 494Unloading Data from Greenplum Database......................................................................... 494Formatting Data Files........................................................................................................... 497Example Custom Data Access Protocol.............................................................................. 500

Managing Performance................................................................................................................... 507Defining Database Performance.......................................................................................... 507Common Causes of Performance Issues............................................................................ 508Greenplum Database Memory Overview............................................................................. 511


6

Managing Resources............................................................................................................515Investigating a Performance Problem.................................................................................. 545

Chapter 5: Greenplum Database Security Configuration Guide......... 548Securing the Database....................................................................................................................549Greenplum Database Ports and Protocols..................................................................................... 550Configuring Client Authentication.................................................................................................... 554

Allowing Connections to Greenplum Database....................................................................554Editing the pg_hba.conf File.................................................................................................556Authentication Methods........................................................................................................ 557SSL Client Authentication.....................................................................................................560PAM-Based Authentication...................................................................................................562Radius Authentication...........................................................................................................563Limiting Concurrent Connections......................................................................................... 564Encrypting Client/Server Connections..................................................................................565

Configuring Database Authorization................................................................................................566Access Permissions and Roles............................................................................................566Managing Object Privileges..................................................................................................566Using SSH-256 Encryption...................................................................................................567Restricting Access by Time..................................................................................................569Dropping a Time-based Restriction.................................................................................... 571

Greenplum Command Center Security........................................................................................... 572Auditing............................................................................................................................................ 575Encrypting Data and Database Connections.................................................................................. 580

Encrypting gpfdist Connections............................................................................................ 580Encrypting Data at Rest with pgcrypto.................................................................................581

Security Best Practices................................................................................................................... 589

Chapter 6: Greenplum Database Best Practices..................................593Best Practices Summary.................................................................................................................594System Configuration...................................................................................................................... 600Schema Design............................................................................................................................... 605

Data Types........................................................................................................................... 605Storage Model...................................................................................................................... 605Compression......................................................................................................................... 606Distributions.......................................................................................................................... 607Partitioning............................................................................................................................ 610Indexes..................................................................................................................................612Column Sequence and Byte Alignment............................................................................... 612

Memory and Resource Management with Resource Groups......................................................... 614Memory and Resource Management with Resource Queues........................................................ 617System Monitoring and Maintenance..............................................................................................621

Monitoring............................................................................................................................. 621Updating Statistics with ANALYZE.......................................................................................622Managing Bloat in a Database.............................................................................................623Monitoring Greenplum Database Log Files..........................................................................627

Loading Data................................................................................................................................... 629INSERT Statement with Column Values..............................................................................629COPY Statement.................................................................................................................. 629External Tables.....................................................................................................................629External Tables with Gpfdist................................................................................................ 629Gpload...................................................................................................................................630Best Practices.......................................................................................................................631

Security............................................................................................................................................ 632


7

Encrypting Data and Database Connections.................................................................................. 635Tuning SQL Queries....................................................................................................................... 644

How to Generate Explain Plans........................................................................................... 644How to Read Explain Plans................................................................................................. 644Optimizing Greenplum Queries............................................................................................ 646

High Availability............................................................................................................................... 648Disk Storage......................................................................................................................... 648Master Mirroring....................................................................................................................648Segment Mirroring................................................................................................................ 649Dual Clusters........................................................................................................................ 650Backup and Restore.............................................................................................................650Detecting Failed Master and Segment Instances................................................................ 651Segment Mirroring Configurations........................................................................................652

Chapter 7: Greenplum Database Utility Guide..................................... 657About the Greenplum Database Utilities.........................................................................................658

Referencing IP Addresses....................................................................................................658Running Backend Server Programs.....................................................................................658

Utility Reference.............................................................................................................................. 660analyzedb..............................................................................................................................660clusterdb................................................................................................................................664createdb................................................................................................................................ 666createlang............................................................................................................................. 668createuser............................................................................................................................. 669dropdb................................................................................................................................... 672droplang................................................................................................................................ 674dropuser................................................................................................................................ 675gpactivatestandby................................................................................................................. 676gpaddmirrors......................................................................................................................... 678gpbackup_manager.............................................................................................................. 681gpbackup...............................................................................................................................685gpcheckcat............................................................................................................................ 691gpcheckperf...........................................................................................................................694gpconfig.................................................................................................................................697gpcopy...................................................................................................................................700gpdeletesystem..................................................................................................................... 700gpexpand.............................................................................................................................. 701gpfdist....................................................................................................................................705gpinitstandby......................................................................................................................... 708gpinitsystem.......................................................................................................................... 710gpload................................................................................................................................... 718gplogfilter...............................................................................................................................729gpmapreduce........................................................................................................................ 731gpmapreduce.yaml................................................................................................................733gpmovemirrors...................................................................................................................... 740gppkg.................................................................................................................................... 741gprecoverseg........................................................................................................................ 743gpreload................................................................................................................................ 747gprestore............................................................................................................................... 749gpscp.....................................................................................................................................755gpssh.....................................................................................................................................757gpssh-exkeys........................................................................................................................ 760gpstart................................................................................................................................... 762gpstate.................................................................................................................................. 765gpstop................................................................................................................................... 769


8

pg_config...............................................................................................................................771pg_dump............................................................................................................................... 773pg_dumpall............................................................................................................................782pg_restore............................................................................................................................. 786pgbouncer............................................................................................................................. 791pgbouncer.ini.........................................................................................................................793pgbouncer-admin.................................................................................................................. 805plcontainer.............................................................................................................................813plcontainer Configuration File...............................................................................................818psql........................................................................................................................................822reindexdb.............................................................................................................................. 847vacuumdb..............................................................................................................................848

Additional Supplied Programs......................................................................................................... 851

Chapter 8: Greenplum Database Reference Guide.............................. 852SQL Commands.............................................................................................................................. 853

SQL Syntax Summary..........................................................................................................856ABORT..................................................................................................................................894ALTER AGGREGATE...........................................................................................................895ALTER COLLATION.............................................................................................................896ALTER CONVERSION......................................................................................................... 897ALTER DATABASE.............................................................................................................. 898ALTER DEFAULT PRIVILEGES.......................................................................................... 900ALTER DOMAIN...................................................................................................................902ALTER EXTENSION.............................................................................................................904ALTER EXTERNAL TABLE..................................................................................................907ALTER FOREIGN DATA WRAPPER...................................................................................908ALTER FOREIGN TABLE.................................................................................................... 910ALTER FUNCTION...............................................................................................................913ALTER GROUP.................................................................................................................... 916ALTER INDEX...................................................................................................................... 917ALTER LANGUAGE............................................................................................................. 918ALTER MATERIALIZED VIEW.............................................................................................919ALTER OPERATOR............................................................................................................. 920ALTER OPERATOR CLASS................................................................................................ 921ALTER OPERATOR FAMILY...............................................................................................922ALTER PROTOCOL............................................................................................................. 924ALTER RESOURCE GROUP.............................................................................................. 925ALTER RESOURCE QUEUE...............................................................................................928ALTER ROLE....................................................................................................................... 930ALTER SCHEMA..................................................................................................................935ALTER SEQUENCE............................................................................................................. 935ALTER SERVER.................................................................................................................. 938ALTER TABLE......................................................................................................................939ALTER TABLESPACE..........................................................................................................952ALTER TEXT SEARCH CONFIGURATION.........................................................................953ALTER TEXT SEARCH DICTIONARY................................................................................ 954ALTER TEXT SEARCH PARSER........................................................................................956ALTER TEXT SEARCH TEMPLATE....................................................................................956ALTER TYPE........................................................................................................................957ALTER USER....................................................................................................................... 960ALTER USER MAPPING..................................................................................................... 961ALTER VIEW........................................................................................................................ 961ANALYZE..............................................................................................................................963BEGIN................................................................................................................................... 967


9

CHECKPOINT.......................................................................................................................969CLOSE.................................................................................................................................. 969CLUSTER............................................................................................................................. 970COMMENT............................................................................................................................972COMMIT................................................................................................................................975COPY.................................................................................................................................... 975CREATE AGGREGATE........................................................................................................987CREATE CAST.....................................................................................................................994CREATE COLLATION..........................................................................................................997CREATE CONVERSION...................................................................................................... 998CREATE DATABASE......................................................................................................... 1000CREATE DOMAIN..............................................................................................................1002CREATE EXTENSION........................................................................................................1003CREATE EXTERNAL TABLE.............................................................................................1005CREATE FOREIGN DATA WRAPPER..............................................................................1014CREATE FOREIGN TABLE............................................................................................... 1016CREATE FUNCTION..........................................................................................................1018CREATE GROUP............................................................................................................... 1028CREATE INDEX................................................................................................................. 1028CREATE LANGUAGE........................................................................................................ 1033CREATE MATERIALIZED VIEW........................................................................................1035CREATE OPERATOR........................................................................................................ 1037CREATE OPERATOR CLASS........................................................................................... 1041CREATE OPERATOR FAMILY..........................................................................................1045CREATE PROTOCOL........................................................................................................ 1046CREATE RESOURCE GROUP......................................................................................... 1047CREATE RESOURCE QUEUE..........................................................................................1050CREATE ROLE.................................................................................................................. 1053CREATE RULE...................................................................................................................1058CREATE SCHEMA.............................................................................................................1060CREATE SEQUENCE........................................................................................................ 1061CREATE SERVER............................................................................................................. 1064CREATE TABLE.................................................................................................................1065CREATE TABLE AS...........................................................................................................1080CREATE TABLESPACE.....................................................................................................1084CREATE TEXT SEARCH CONFIGURATION....................................................................1085CREATE TEXT SEARCH DICTIONARY............................................................................1086CREATE TEXT SEARCH PARSER...................................................................................1087CREATE TEXT SEARCH TEMPLATE...............................................................................1088CREATE TYPE...................................................................................................................1089CREATE USER.................................................................................................................. 1097CREATE USER MAPPING................................................................................................ 1098CREATE VIEW................................................................................................................... 1099DEALLOCATE.................................................................................................................... 1101DECLARE........................................................................................................................... 1102DELETE.............................................................................................................................. 1104DISCARD............................................................................................................................ 1107DO.......................................................................................................................................1108DROP AGGREGATE..........................................................................................................1110DROP CAST.......................................................................................................................1111DROP COLLATION............................................................................................................ 1112DROP CONVERSION........................................................................................................ 1112DROP DATABASE............................................................................................................. 1113DROP DOMAIN.................................................................................................................. 1114DROP EXTENSION............................................................................................................1115DROP EXTERNAL TABLE.................................................................................................1115


10

DROP FOREIGN DATA WRAPPER..................................................................................1116DROP FOREIGN TABLE................................................................................................... 1117DROP FUNCTION..............................................................................................................1118DROP GROUP................................................................................................................... 1119DROP INDEX..................................................................................................................... 1119DROP LANGUAGE.............................................................................................................1120DROP MATERIALIZED VIEW............................................................................................1121DROP OPERATOR............................................................................................................ 1122DROP OPERATOR CLASS............................................................................................... 1123DROP OPERATOR FAMILY.............................................................................................. 1123DROP OWNED...................................................................................................................1124DROP PROTOCOL............................................................................................................ 1125DROP RESOURCE GROUP..............................................................................................1126DROP RESOURCE QUEUE.............................................................................................. 1127DROP ROLE.......................................................................................................................1128DROP RULE.......................................................................................................................1129DROP SCHEMA................................................................................................................. 1130DROP SEQUENCE............................................................................................................ 1130DROP SERVER..................................................................................................................1131DROP TABLE..................................................................................................................... 1132DROP TABLESPACE.........................................................................................................1133DROP TEXT SEARCH CONFIGURATION........................................................................1134DROP TEXT SEARCH DICTIONARY................................................................................1134DROP TEXT SEARCH PARSER....................................................................................... 1135DROP TEXT SEARCH TEMPLATE...................................................................................1136DROP TYPE....................................................................................................................... 1137DROP USER...................................................................................................................... 1137DROP USER MAPPING.....................................................................................................1138DROP VIEW....................................................................................................................... 1139END.....................................................................................................................................1139EXECUTE........................................................................................................................... 1140EXPLAIN............................................................................................................................. 1141FETCH................................................................................................................................ 1146GRANT................................................................................................................................1148INSERT............................................................................................................................... 1153LOAD.................................................................................................................................. 1156LOCK.................................................................................................................................. 1157MOVE..................................................................................................................................1160PREPARE........................................................................................................................... 1161REASSIGN OWNED...........................................................................................................1163REFRESH MATERIALIZED VIEW..................................................................................... 1164REINDEX............................................................................................................................ 1165RELEASE SAVEPOINT......................................................................................................1167RESET................................................................................................................................ 1168REVOKE............................................................................................................................. 1168ROLLBACK......................................................................................................................... 1171ROLLBACK TO SAVEPOINT.............................................................................................1172SAVEPOINT........................................................................................................................1173SELECT.............................................................................................................................. 1175SELECT INTO.................................................................................................................... 1193SET..................................................................................................................................... 1194SET CONSTRAINTS.......................................................................................................... 1196SET ROLE.......................................................................................................................... 1197SET SESSION AUTHORIZATION..................................................................................... 1198SET TRANSACTION.......................................................................................................... 1199SHOW................................................................................................................................. 1202


11

START TRANSACTION..................................................................................................... 1203TRUNCATE.........................................................................................................................1204UPDATE..............................................................................................................................1206VACUUM.............................................................................................................................1209VALUES.............................................................................................................................. 1212

Data Types.................................................................................................................................... 1215Date/Time Types................................................................................................................ 1217Pseudo-Types..................................................................................................................... 1227Text Search Data Types.................................................................................................... 1229Range Types...................................................................................................................... 1231

Summary of Built-in Functions...................................................................................................... 1236Greenplum Database Function Types................................................................................1236Built-in Functions and Operators........................................................................................1237JSON Functions and Operators......................................................................................... 1240Window Functions.............................................................................................................. 1247Advanced Aggregate Functions......................................................................................... 1249Text Search Functions and Operators............................................................................... 1251Range Functions and Operators........................................................................................ 1255

Additional Supplied Modules......................................................................................................... 1258auto_explain........................................................................................................................1258citext....................................................................................................................................1258dblink...................................................................................................................................1259diskquota.............................................................................................................................1262fuzzystrmatch...................................................................................................................... 1266gp_sparse_vector................................................................................................................1267hstore.................................................................................................................................. 1271orafce.................................................................................................................................. 1271pageinspect.........................................................................................................................1273pgcrypto.............................................................................................................................. 1273sslinfo.................................................................................................................................. 1274

Character Set Support...................................................................................................................1275Setting the Character Set...................................................................................................1277Character Set Conversion Between Server and Client...................................................... 1277

Server Configuration Parameters..................................................................................................1280Parameter Types and Values.............................................................................................1280Setting Parameters............................................................................................................. 1280Parameter Categories.........................................................................................................1281Configuration Parameters...................................................................................................1291

System Catalogs........................................................................................................................... 1373System Tables.................................................................................................................... 1373System Views..................................................................................................................... 1374System Catalogs Definitions...............................................................................................1375

The gp_toolkit Administrative Schema..........................................................................................1476Checking for Tables that Need Routine Maintenance........................................................1476Checking for Locks.............................................................................................................1477Checking Append-Optimized Tables.................................................................................. 1479Viewing Greenplum Database Server Log Files................................................................ 1483Checking Server Configuration Files..................................................................................1486Checking for Failed Segments........................................................................................... 1487Checking Resource Group Activity and Status.................................................................. 1488Checking Resource Queue Activity and Status................................................................. 1492Checking Query Disk Spill Space Usage...........................................................................1494Viewing Users and Groups (Roles)....................................................................................1496Checking Database Object Sizes and Disk Space............................................................ 1496Checking for Uneven Data Distribution.............................................................................. 1500Including Data for Materialized Views................................................................................ 1501


12

The gpperfmon Database..............................................................................................................1503database_*.........................................................................................................................1505diskspace_*....................................................................................................................... 1506interface_stats_*................................................................................................................ 1506log_alert_*..........................................................................................................................1508queries_*............................................................................................................................. 1509segment_*..........................................................................................................................1511socket_stats_*.....................................................................................................................1512system_*............................................................................................................................. 1513dynamic_memory_info........................................................................................................ 1515memory_info...................................................................................................................... 1515

Server Programmatic Interfaces....................................................................................................1517Greenplum Partner Connector API.................................................................................... 1517Developing a Background Worker Process....................................................................... 1536

SQL Features, Reserved and Key Words, and Compliance........................................................ 1539Summary of Greenplum Features...................................................................................... 1539Reserved Identifiers and SQL Key Words......................................................................... 1549SQL 2008 Optional Feature Compliance........................................................................... 1565

Chapter 9: Greenplum Client and Loader Tools Package................. 1595

Chapter 10: About the Tools Package................................................ 1596

Chapter 11: Installing the Client and Loader Tools Package............ 1597Supported Platforms......................................................................................................................1598Installation Procedure....................................................................................................................1599About Your Installation.................................................................................................................. 1600Running the UNIX Tools Installer................................................................................................. 1601

Prerequisites....................................................................................................................... 1601Procedure............................................................................................................................1601

Running the Windows Tools Installer........................................................................................... 1602Prerequisites....................................................................................................................... 1602Procedure............................................................................................................................1602

Chapter 12: Configuring Greenplum Database for Remote ClientAccess................................................................................................. 1603

Chapter 13: Configuring a Client System for KerberosAuthentication.................................................................................... 1604

Chapter 14: Using the Client and Loader Tools................................. 1605Prerequisites.................................................................................................................................. 1606Setting Up Your Greenplum Database Clients Runtime Environment.......................................... 1607Running the Client and Loader Programs.................................................................................... 1608Greenplum Database Documentation References........................................................................1609Windows Considerations............................................................................................................... 1610

Chapter 15: Client and Loader Utility Reference................................1611


13

Chapter 16: DataDirect ODBC Drivers for Pivotal Greenplum...........1612Prerequisites.................................................................................................................................. 1613Supported Client Platforms........................................................................................................... 1614Installing on Linux Systems.......................................................................................................... 1615

Configuring the Driver on Linux......................................................................................... 1616Testing the Driver Connection on Linux.............................................................................1617

Installing on Windows Systems.................................................................................................... 1619Verifying the Version on Windows..................................................................................... 1619Configuring and Testing the Driver on Windows................................................................1619

DataDirect Driver Documentation..................................................................................................1621

Chapter 17: DataDirect JDBC Driver for Pivotal Greenplum............. 1622Prerequisites.................................................................................................................................. 1623Downloading the DataDirect JDBC Driver.................................................................................... 1624Obtaining Version Details for the Driver....................................................................................... 1625Usage Information......................................................................................................................... 1626Configuring Prepared Statement Execution..................................................................................1627DataDirect Driver Documentation..................................................................................................1628

Pivotal Greenplum 6.8 Release Notes Release Notes

14

Chapter 2

Pivotal Greenplum 6.8 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.8 releases.For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum DatabaseDocumentation. For information about Greenplum Database end of life, see Pivotal Greenplum Databaseend of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on PivotalNetwork.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of GreenplumDatabase. Only Pivotal Greenplum Database is supported by Pivotal Support.

https://gpdb.docs.pivotal.io/https://gpdb.docs.pivotal.io/https://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://support.pivotal.io/hc/en-us/articles/201143553-GPDB-software-EOL-policyhttps://network.pivotal.io/products/pivotal-gpdbhttps://network.pivotal.io/products/pivotal-gpdbhttp://greenplum.org/


15

Release 6.8.1Release Date: 2020-06-11

Pivotal Greenplum 6.8.1 is a maintenance release that resolves several issues.

Resolved IssuesPivotal Greenplum 6.8.1 resolves these issues:

30684 - Query Optimizer

GPORCA returned incorrect results for some queries when the query's select list containsa window function and the window function contains a correlated subquery or an outerreference. Now the query falls back to the Postgres planner.


GPORCA query performance degraded when compared with Greenplum 5 for somequeries that perform joins using an equality predicate and the equality predicate containsa function, for example coalesce(tbl1.a, '999999') = coalesce(tbl2.a,'999999'). The performance issue was caused by inaccurate cardinality estimates.GPORCA cardinality estimation has been improved for the specified type of query.

172732495, 9953 - query execution

Greenplum Database generated a PANIC when executing a query that executes multipleuser-defined functions and more than one of the functions is defined with the EXECUTE ONINITPLAN attribute. This issue is resolved.

172098556 - psql

Resolved a problem where the psql client \dm command did not display materializedviews.

172094194, 9837 - gprecoverseg

In some cases when recovering segment instances using the gprecoverseg utility withthe -i option to specify details about failed segments torecover, the utility changed some segment instance dbid values in the Greenplum systemconfiguration. This issue is resolved.

https://github.com/greenplum-db/gpdb/issues/9953https://github.com/greenplum-db/gpdb/issues/9837


16

Release 6.8.0Release Date: 2020-06-05

Pivotal Greenplum 6.8.0 is a minor release that includes changed features and resolves several issues.

FeaturesGreenplum Database 6.8.0 includes these new and changed features:

• Greenplum Streaming Server (GPSS) version 1.3.6 is included, which introduces many new andchanged features and bug fixes since the last GPSS version installed in Greenplum 6.x (1.3.1).Refer to the GPSS Release Notes for more information on release content and to access the GPSSdocumentation.

Note: If you have previously used GPSS in your Greenplum 6.x installation, you are required toperform upgrade actions as described in Upgrading the Streaming Server.

• The gpinitsystem input configuration file specified with the -I option supports an additional format tospecify hosts. The QD_PRIMARY_ARRAY, PRIMARY_ARRAY, and MIRROR_ARRAY host parameters maynow be specified using either of the following formats:

host~port~data_directory/seg_prefix~dbid~content_id

hostname~address~port~data_directory/seg_prefix~dbid~content_id

The first format, which is the pre-existing format, sets both the hostname and address columns ofthe gp_segment_configuration catalog table to the value in the host field. The second formatsets the hostname and address columns of the gp_segment_configuration catalog table to thevalues in the respective hostname and address fields. See

• PXF version 5.12.0 is included, which introduces new and changed features and bug fixes. See PXFVersion 5.12.0 below.

• PL/Container version 2.1.2 is included, which introduces the following new features:

• Support for R version 3.6.3.• A new --use_local_copy option to the plcontainer add-image command that you can use

to install the specified image only on the local host.• Greenplum Database 6.8 adds support for Moving a Query to a Different Resource Group.• Greenplum Database 6.8 includes a new metrics collector extension that is compatible with Greenplum

Command Center 6.2 and above. If you are using Command Center 6.0 or 6.1 you must download andinstall Command Center 6.2 after you install Greenplum Database 6.8.

PXF Version 5.12.0PXF includes the following new and changed features:

• PXF trims right-padded white space added by Greenplum before it writes Parquet data.• PXF bundles newer hive, jackson-databind, and supporting internal libraries.• A PXF server running on Java 11 can now read from Hive using an external table that specifies a

Hive* profile.• PXF introduces the new custom option IGNORE_MISSING_PATH for external tables that you use to

read file-based data. Setting this option may be useful when a PXF external table is a child partition of apartitioned Greenplum table. Refer to About PXF External Table Child Partitions for more information.

• PXF bundles the jodd-core library to satisfy a missing transitive dependency that is required whenPXF reads Parquet files that contain data in timestamp format.

https://greenplum.docs.pivotal.io/streaming-server/1-3-6/gpss-v13-release-notes.htmlhttps://greenplum.docs.pivotal.io/streaming-server/1-3-6/upgrading-gpss.html../pxf/troubleshooting_pxf.html#pxf-tblpart


17

• PXF adds column projection support for the Hive and HiveRC profiles by changing the implementationto use column name-based, rather than column index-based, mapping.

Note: If you have existing PXF external tables that specify a Hive* profile, you may berequired to perform upgrade actions as described in Upgrading PXF.

Resolved IssuesPivotal Greenplum 6.8.0 resolves these issues:

329, 30602 - PXF

PXF did not correctly read a partitioned Hive table when the external table specified aHive* profile and the external table and Hive table had a differing number of columns.This issue is resolved. PXF now supports column projection for the Hive* profiles andcorrectly handles this situation.


When falling back to the Postgres planner, GPORCA incorrectly logged messages thatwere internal messages. This made the log file difficult to read and caused bloat in the file.This issue is resolved. GPORCA message logging has been improved and the internalmessages are no longer sent to the log files.

30585 - Locking

Resolved a problem that could corrupt resource queue locks, and potentially other types oflocks, in shared memory. This problem could cause errors such as lock lock_name onobject object_identifier is already held.

30557 - DDL

When performing a data reorganization with the ALTER TABLE command on a leafpartition of a partitioned table that did not change the distribution policy, GreenplumDatabase returned the error ERROR: can't set the distribution policy.This type of redistribution is allowed in Greenplum 5. Now Greenplum allows datareorganization on a leaf partition if the distribution policy is not changed.


When GPORCA performed dynamic partition elimination for some queries againstpartitioned tables that perform joins, GPORCA was not using the correct statistics. Thiscaused a performance degradation when compared with Greenplum 5. GPORCA hasimproved how statistics are computed tor the specified type of query.

172854840 - Interconnect

In some cases, a query that executes a stable function that contains an SQL statementmight hang because the query dispatcher (QD) did not correctly manage the execution ofthe function and the dispatching of the query plan. This issue is resolved.

172832212 - Interconnect

In some cases, communication between a query dispatcher (QD) and a query executor(QE) on different segments was slow when the Greenplum interconnect type is set tothe TCP networking protocol for Greenplum Database interconnect traffic. Now thecommunication between a QD and a QE is more efficient.


For text data types, the GPORCA the cardinality estimation algorithm has been improvedfor equality comparisons. For example, when a query contains an IN clause that containstext elements.

172576000 - COPY

../pxf/upgrade_pxf_6x.htmlhttps://github.com/greenplum-db/pxf/issues/329


18

If data format errors occurred while copying data into a partitioned table with a COPY FROMcommand in single row error isolation mode, Greenplum Database might crash when aquery executor (QE) did not handle the data format error correctly. This issue is resolved.

30487 - Utility Commands

On a Greenplum Database 6 system with FIPS enabled, Greenplum utility commands suchas gpinitsystem returned the error "ERROR:root:code for hash md5 was notfound." This issue is resolved.

30484 - Utility Commands

When initializing a Greenplum Database system with gpinitsystem, the primarysegments were erroneously named using DNS resolvable external hostnames instead ofthe internal interconnect interface hostnames. At the same time, the segment mirrors werecorrectly named. This issue is now resolved.

Upgrading from Greenplum 6.x to Greenplum 6.8Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software toGreenplum 6.8.0.


19

Deprecated FeaturesDeprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum6.x deprecates:

• The gpsys1 utility.• The analzyedb option --skip_root_stats (deprecated since 6.2).

If the option is specified, a warning is issued stating that the option will be ignored.• The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).• The following PXF configuration properties (deprecated since 6.2):

• The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settingsfor your new Hadoop server configurations.

• The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use thepxf.service.user.impersonation property to configure user impersonation for a new JDBCserver configuration.

• The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

To avoid a Greenplum Database syntax error, set the value of this parameter to true when yourun applications that execute CREATE EXTERNAL TABLE or COPY commands that include the nowremoved Greenplum Database 4.3.x INTO ERROR TABLE clause.

• Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).• The Greenplum external table C API (deprecated since 6.0).

Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.• Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION

BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command(deprecated since 6.0).

Using this undocumented syntax will generate a deprecation warning message.• The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

This format could not be parsed unambiguously in previous Greenplum Database releases, and is notsupported in PostgreSQL 9.4.

• The createlang and droplang utilities (deprecated since 6.0).• The pg_resqueue_status system view (deprecated since 6.0).

Use the gp_toolkit.gp_resqueue_status view instead.• The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and

CREATE TABLE AS commands (deprecated since 6.0).

These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.• The Greenplum Platform Extension Framework (PXF) HDFS profile names for the Text, Avro, JSON,

Parquet, and SequenceFile data formats (deprecated since 5.16).

Refer to Connectors, Data Formats, and Profiles in the PXF Hadoop documentation for moreinformation.

• Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table(deprecated since 6.0).

• Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecatedsince 6.0).

This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

• The @@@ text search operator (deprecated since 6.0).

../pxf/access_hdfs.html#hadoop_connectors


20

This operator is currently a synonym for the @@ operator.• The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

This syntax requires that the options to the command be specified in a specific order.• The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).


21

Migrating Data to Greenplum 6Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases,or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existingGreenplum data to Greenplum 6, using standard backup and restore procedures.


22

Known Issues and LimitationsPivotal Greenplum 6 has these limitations:

• Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6is not supported.

• MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.• Greenplum 6 is not supported for installation on DCA systems.• Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1: Key Known Issues in Pivotal Greenplum 6.x

Issue Category Description

N/A PXF Starting in 6.x, Greenplum does not bundle cURL and instead loadsthe system-provided library. PXF requires cURL version 7.29.0 ornewer. The officially-supported cURL for the CentOS 6.x and Red HatEnterprise Linux 6.x operating systems is version 7.19.*. GreenplumDatabase 6 does not support running PXF on CentOS 6.x or RHEL 6.x due to this limitation.

Workaround: Upgrade the operating system of your GreenplumDatabase 6 hosts to CentOS 7+ or RHEL 7+, which provides a cURLversion suitable to run PXF.

29703 Loading Datafrom ExternalTables

Due to limitations in the Greenplum Database external tableframework, Greenplum Database cannot log the following types oferrors that it encounters while loading data:

• data type parsing errors• unexpected value type errors• data type conversion errors• errors returned by native and user-defined functions

LOG ERRORS returns error information for data exceptions only.When it encounters a parsing error, Greenplum terminates the loadjob, but it cannot log and propagate the error back to the user via gp_read_error_log().

Workaround: Clean the input data before loading it into GreenplumDatabase.

30594 ResourceManagement

Resource queue-related statistics may be inaccurate in certain cases.Pivotal recommends that you use the resource group resourcemanagement scheme that is available in Greenplum 6.

30522 Logging Greenplum Database may write a FATAL message to the standbymaster or mirror log stating that the database system is in recoverymode when the instance is synchronizing with the master andGreenplum attempts to contact it before the operation completes.Ignore these messages and use gpstate -f output to determine ifthe standby successfully synchronized with the Greenplum master;the command returns Sync state: sync if it is synchronized.


23

Issue Category Description

30537 PostgresPlanner

The Postgres Planner generates a very large query plan that causesout of memory issues for the following type of CTE (common tableexpression) query: the WITH clause of the CTE contains a partitionedtable with a large number partitions, and the WITH reference is usedin a subquery that joins another partitioned table.

Workaround: If possible, use the GPORCA query optimizer. Withthe server configuration parameter optimizer=on, GreenplumDatabase attempts to use GPORCA for query planning andoptimization when possible and falls back to the Postgres Plannerwhen GPORCA cannot be used. Also, the specified type of querymight require a long time to complete.

170824967 gpfidsts For Greenplum Database 6.x, a command that accesses an externaltable that uses the gpfdists protocol fails if the external tabledoes not use an IP address when specifying a host system in theLOCATION clause of the external table definition.

n/a MaterializedViews

By default, certain gp_toolkit views do not display data formaterialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internalview as described in Including Data for Materialized Views.

168957894 PXF The PXF Hive Connector does not support using the Hive* profilesto access Hive transactional tables.

Workaround: Use the PXF JDBC Connector to access Hive.

168548176 gpbackup When using gpbackup to back up a Greenplum Database 5.7.1 orearlier 5.x release with resource groups enabled, gpbackup returnsa column not found error for t6.value AS memoryauditor.

164791118 PL/R PL/R cannot be installed using the deprecated createlang utility,and displays the error:

createlang: language installation failed: ERROR:no schema has been selected to create in

Workaround: Use CREATE EXTENSION to install PL/R, as describedin the documentation.

N/A GreenplumClient/LoadTools onWindows

Documents

Pivotal Greenplum Database · Contents Release Notes 4 Initializing a Greenplum Database System.....62