Pivotal Greenplum 6.10 Release Notes

Pivotal Greenplum 6.10 Release Notes

This document contains pertinent release information about Pivotal Greenplum Database 6.10 releases. For previous versions of the release notes for Greenplum Database, go to Pivotal Greenplum Database Documentation. For information about Greenplum Database end of life, see Pivotal Greenplum Database end of life policy.

Pivotal Greenplum 6 software is available for download from the Pivotal Greenplum page on Pivotal Network.

Pivotal Greenplum 6 is based on the open source Greenplum Database project code.

Important: Pivotal Support does not provide support for open source versions of Greenplum Database. Only Pivotal Greenplum Database is supported by Pivotal Support.

Release 6.10.0

Release Date: 2020-08-07

Pivotal Greenplum 6.10.0 is a minor release that includes changed features and resolves several issues.

Features

Greenplum Database 6.10.0 includes these new and changed features:

  • Greenplum Database 6.10 introduces the server configuration parameter max_slot_wal_keep_size that sets the maximum size in megabytes of replication WAL log files on disk per segment instance. The default is -1, Greenplum can retain an unlimited amount of WAL files on disk.
  • Greenplum Database 6.10 introduces the server configuration parameter gp_add_column_inherits_table_setting for append-optimized, column-oriented tables. When adding a column to a table with the ALTER TABLE command, the parameter controls whether the table's data compression parameters for a column (compresstype, compresslevel, and blocksize) can be inherited from WITH clause values when the table was created. The default is off, the table's data compression settings are not considered when adding a column to the table. If the value is on, the table's WITH clause values are considered.
  • When reading data, the gpload utility now supports the control file parameter FILL_MISSING_FIELDS that can add NULL values to a data row if the row has trailing field values that are missing. To enable this feature, set the parameter to true.
  • Greenplum Database supports using proxies for Greenplum Database interconnect communication to reduce the use of connections and ports during query processing. The interconnect proxies consumes fewer connections and ports than TCP mode, and has better performance than UDPIFC mode in a high-latency network.
    To enable interconnect proxies for the Greenplum system, set these system configuration parameters.
    • List the proxy ports with the new parameter gp_interconnect_proxy_addresses. You must specify a proxy port for the master, standby master, and all segment instances.
    • Set the parameter gp_interconnect_type to proxy. The proxy value is new in Greenplum Database 6.10.
  • The PgBouncer connection pooler was updated from version 1.8.1 to version 1.13. To support this change on RHEL/CentOS 6 platforms, the Greenplum Database package now requires libevent2 instead of libevent. RHEL/CentOS 7 requirements are unchanged. See the PgBouncer 1.13.x Release Notes for a summary of changes.
  • The gpcheckcat catalog verification utility adds a new test, aoseg_table, that you can use to check that the vertical partition information on append-optimized, column storage tables is consistent with pg_attribute.
  • Greenplum Database 6.10 introduces a new server configuration parameter, gp_fts_replication_attempt_count, that you can use to configure the maximum number of times that the Greenplum fault tolerance service (FTS) attempts to establish a primary-mirror replication connection.
  • PXF version 5.14.0 is included, which includes new and changed features and bug fixes. Refer to the PXF Release Notes for more information on release content and supported platforms, and to access the PXF documentation.
  • Greenplum Streaming Server (GPSS) version 1.4.1 is included, which includes changes and bug fixes. Refer to the GPSS Release Notes for more information on release content and to access the GPSS documentation.
    Note: If you have previously used GPSS in your Greenplum 6.x installation, you are required to perform upgrade actions as described in Upgrading the Streaming Server.

Resolved Issues

Pivotal Greenplum 6.10.0 resolves these issues:

30554 - ANALYZE, gprestore
In some cases, when performing concurrent ANALYZE operations on large partition tables generated the error Canceling query because of high VMEM usage. This issue occurred during some restore operations with gprestore. This issue is resolved.
30583 - Transaction Management
In some cases, a segment mirror went offline when the replication WAL log files on the mirror system caused disk full issues. Greenplum Database introduces the server configuration parameter max_slot_wal_keep_size to limit the amount of WAL logs stored on disk. See Features.
30792 - Logging
Resolves a disk space issue encountered when a query that generated a large number of spill files also generated an excessive number of HashJoin: Too many batches computed log messages by decreasing the severity level of the message.
173680224 - gprecoverseg
In some cases, gprecoverseg hangs when performing an incremental recovery while trying to perform a clean shutdown of the segment instance due to a locking issue. This issue is resolved..
30781, 9427 - Postgres Planner
For some queries that perform joins between partitioned tables, Greenplum Database returned ERROR: unrecognized path type 106. This issue is resolved.
10419 - Postgres Planner
In some cases when generating query plans, the Postgres Planner did not handle volatile functions correctly and allowed multiple executions of the function during query execution. This could cause incorrect results. This issue is resolved.
30692 - System Catalog Functions
pg_get_viewdef() returned an incorrect view definition when the view was created with the Greenplum-specific CASE WHEN (arg1) IS NOT DISTINCT FROM (arg2) clause. This issue is resolved.
30558 - Query Optimizer
Resolves an issue where execution time and spill file size increased for a query on a larger width table because Greenplum Database overestimated row cardinality when the query specified multiple predicates that included distribution keys.
30512 - MPP: Dispatch
Resolves an issue where Greenplum Database hung while continuously retrying a primary-mirror replication connection. Greenplum 6.10 introduces a new server configuration parameter, gp_fts_replication_attempt_count, with which you can configure the maximum number of retry attempts.
10230 - Server: Execution
Greenplum Database generated a segmentation fault during execution of a hash aggregate query when it incorrectly initialized a tuple memory context for a partitioned table that contained a btree index. This issue is resolved.
10141 - ALTER TABLE ... INHERIT
Greenplum Database restricted you from creating both a replicated table that inherits from another table, and a table that inherits from a replicated table, but Greenplum did allow you to use the ALTER TABLE ... INHERIT command to assign inheritance to/from a replicated table after it was created. Update commands on such a table returned the misleading error ModifyTable mixes distributed and entry-only tables. This issue is resolved; Greenplum 6.10 enforces replicated table inheritance restrictions uniformly, including on ALTER TABLE ... INHERIT statements.
10057 - Server: Transactions
Greenplum Database returned the error Too many distributed transactions for snapshot when processing many distributed transactions and the max_prepared_transactions server configuration parameter was set to a low value. This issue is resolved; Greenplum now uses a more robust method to determine the maximum number of distributed transactions.
10030 - Server: Query Execution
Greenplum Database incorrectly used the value of the gp_enable_global_deadlock_detector server configuration parameter on query executors to determine the lock mode. This issue is resolved, the parameter is now master only.
9896 - gpexpand
gpexpand failed on a partitioned Greenplum Database table that included an external table leaf child partition. This issue is resolved; Greenplum Database now skips external table partitions during expansion.
9207 - Server: Transactions
In certain situations, the Greenplum Database standby master host shutdown down and exited abnormally with the FATAL error the limit of N distributed transactions has been reached when the global transaction limit was exceeded. This issue is resolved; Greenplum now uses a more robust method to determine the maximum number of distributed transactions.
173219210 - Resource Groups
The Greenplum Database gp_toolkit.gp_resgroup_status_per_host view incorrectly reported CPU usage as the sum of resource group CPU usage on all segments on the same host. This issue is resolved; Greenplum Database now correctly reports the average CPU usage of resource groups on all segments on the host.
171849582 - Tablespace
Greenplum Database generated a PANIC when it did not check for the existence of a tablespace directory before attempting to delete the directory. This issue is resolved.

Upgrading from Greenplum 6.x to Greenplum 6.10

Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases, or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software to Greenplum 6.10.0.

Deprecated Features

Deprecated features will be removed in a future major release of Greenplum Database. Pivotal Greenplum 6.x deprecates:

  • The gpsys1 utility.
  • The analzyedb option --skip_root_stats (deprecated since 6.2).

    If the option is specified, a warning is issued stating that the option will be ignored.

  • The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).
  • The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

    To avoid a Greenplum Database syntax error, set the value of this parameter to true when you run applications that execute CREATE EXTERNAL TABLE or COPY commands that include the now removed Greenplum Database 4.3.x INTO ERROR TABLE clause.

  • Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).
  • The Greenplum external table C API (deprecated since 6.0).

    Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.

  • Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command (deprecated since 6.0).

    Using this undocumented syntax will generate a deprecation warning message.

  • The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

    This format could not be parsed unambiguously in previous Greenplum Database releases, and is not supported in PostgreSQL 9.4.

  • The createlang and droplang utilities (deprecated since 6.0).
  • The pg_resqueue_status system view (deprecated since 6.0).

    Use the gp_toolkit.gp_resqueue_status view instead.

  • The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and CREATE TABLE AS commands (deprecated since 6.0).

    These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.

  • Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table (deprecated since 6.0).
  • Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecated since 6.0).

    This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

  • The @@@ text search operator (deprecated since 6.0).

    This operator is currently a synonym for the @@ operator.

  • The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

    This syntax requires that the options to the command be specified in a specific order.

  • The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).

Migrating Data to Greenplum 6

Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases, or from earlier Greenplum 6 Beta releases.

See Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existing Greenplum data to Greenplum 6, using standard backup and restore procedures.

Known Issues and Limitations

Pivotal Greenplum 6 has these limitations:

  • Upgrading a Greenplum Database 4 or 5 release, or Greenplum 6 Beta release, to Pivotal Greenplum 6 is not supported.
  • MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.
  • Greenplum 6 is not supported for installation on DCA systems.
  • Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in Pivotal Greenplum 6.x.

Table 1. Key Known Issues in Pivotal Greenplum 6.x
Issue Category Description
N/A Backup/Restore Restoring the Greenplum Database backup for a table fails in Greenplum 6 versions earlier than version 6.10 when a replicated table has an inheritance relationship to/from another table that was assigned via an ALTER TABLE ... INHERIT statement after table creation.
Workaround: Use the following SQL commands to determine if Greenplum Database includes any replicated tables that inherit from a parent table, or if there are replicated tables that are inherited by a child table: 
SELECT inhrelid::regclass FROM pg_inherits,
  gp_distribution_policy dp
WHERE inhrelid=dp.localoid AND dp.policytype='r';
SELECT inhparent::regclass FROM pg_inherits,
  gp_distribution_policy dp
WHERE inhparent=dp.localoid AND dp.policytype='r';

If these queries return any tables, you may choose to run gprestore with the -–on-error-continue flag to not fail the entire restore when this issue is hit. Or, you can specify the list of tables returned by the queries to the -–exclude-table-file option to skip those tables during restore. You must recreate and repopulate the affected tables after restore.
N/A Spark Connector This version of Greenplum is not compatible with Greenplum-Spark Connector versions earlier than version 1.7.0, due to a change in how Greenplum handles distributed transaction IDs.
N/A PXF Starting in 6.x, Greenplum does not bundle cURL and instead loads the system-provided library. PXF requires cURL version 7.29.0 or newer. The officially-supported cURL for the CentOS 6.x and Red Hat Enterprise Linux 6.x operating systems is version 7.19.*. Greenplum Database 6 does not support running PXF on CentOS 6.x or RHEL 6.x due to this limitation.

Workaround: Upgrade the operating system of your Greenplum Database 6 hosts to CentOS 7+ or RHEL 7+, which provides a cURL version suitable to run PXF.
29703 Loading Data from External Tables Due to limitations in the Greenplum Database external table framework, Greenplum Database cannot log the following types of errors that it encounters while loading data:
  • data type parsing errors
  • unexpected value type errors
  • data type conversion errors
  • errors returned by native and user-defined functions
LOG ERRORS returns error information for data exceptions only. When it encounters a parsing error, Greenplum terminates the load job, but it cannot log and propagate the error back to the user via gp_read_error_log().

Workaround: Clean the input data before loading it into Greenplum Database.
30594 Resource Management Resource queue-related statistics may be inaccurate in certain cases. Pivotal recommends that you use the resource group resource management scheme that is available in Greenplum 6.
30522 Logging Greenplum Database may write a FATAL message to the standby master or mirror log stating that the database system is in recovery mode when the instance is synchronizing with the master and Greenplum attempts to contact it before the operation completes. Ignore these messages and use gpstate -f output to determine if the standby successfully synchronized with the Greenplum master; the command returns Sync state: sync if it is synchronized.
30537 Postgres Planner The Postgres Planner generates a very large query plan that causes out of memory issues for the following type of CTE (common table expression) query: the WITH clause of the CTE contains a partitioned table with a large number partitions, and the WITH reference is used in a subquery that joins another partitioned table.

Workaround: If possible, use the GPORCA query optimizer. With the server configuration parameter optimizer=on, Greenplum Database attempts to use GPORCA for query planning and optimization when possible and falls back to the Postgres Planner when GPORCA cannot be used. Also, the specified type of query might require a long time to complete.
170824967 gpfidsts For Greenplum Database 6.x, a command that accesses an external table that uses the gpfdists protocol fails if the external table does not use an IP address when specifying a host system in the LOCATION clause of the external table definition.
n/a Materialized Views By default, certain gp_toolkit views do not display data for materialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internal view as described in Including Data for Materialized Views.
168957894 PXF The PXF Hive Connector does not support using the Hive* profiles to access Hive transactional tables.

Workaround: Use the PXF JDBC Connector to access Hive.
168548176 gpbackup When using gpbackup to back up a Greenplum Database 5.7.1 or earlier 5.x release with resource groups enabled, gpbackup returns a column not found error for t6.value AS memoryauditor.
164791118 PL/R PL/R cannot be installed using the deprecated createlang utility, and displays the error:
createlang: language installation failed: ERROR:
no schema has been selected to create in
Workaround: Use CREATE EXTENSION to install PL/R, as described in the documentation.
N/A Greenplum Client/Load Tools on Windows The Greenplum Database client and load tools on Windows have not been tested with Active Directory Kerberos authentication.

Differences Compared to Open Source Greenplum Database

Pivotal Greenplum 6.x includes all of the functionality in the open source Greenplum Database project and adds:
  • Product packaging and installation script
  • Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of Greenplum Database due to licensing restrictions.
  • Support for data connectors:
    • Greenplum-Spark Connector
    • Greenplum-Informatica Connector
    • Greenplum-Kafka Integration
    • Greenplum Streaming Server
  • Data Direct ODBC/JDBC Drivers
  • gpcopy utility for copying or migrating objects between Greenplum systems
  • Support for managing Greenplum Database using Pivotal Greenplum Command Center
  • Support for full text search and text analysis using Pivotal GPText
  • Greenplum backup plugin for DD Boost
  • Backup/restore storage plugin API