VMware Tanzu Greenplum 6.16 Release Notes

VMware Tanzu Greenplum 6.16 Release Notes

This document contains pertinent release information about VMware Tanzu Greenplum Database 6.16 releases. For previous versions of the release notes for Greenplum Database, go to VMware Tanzu Greenplum Database Documentation. For information about Greenplum Database end of life, see VMware Tanzu Greenplum Database end of life policy.

VMware Tanzu Greenplum 6 software is available for download from the VMware Tanzu Greenplum page on VMware Tanzu Network.

VMware Tanzu Greenplum 6 is based on the open source Greenplum Database project code.

Release 6.16.3

Release Date: 2021-07-02

VMware Tanzu Greenplum 6.16.3 is a maintenance release that resolves several issues.

Resolved Issues

VMware Tanzu Greenplum 6.16.3 resolves these issues:

Server

31549, 31577
Resolved an issue that made Greenplum unable to cast unknown-type literals to cstring in certain circumstances. This problem could surface in several ways including query panics, out of memory conditions, and the following compression error:
"ERROR","XX000","compressed data is corrupt (pg_lzcompress.c:737)"

Analyze

31589
When running ANALYZE on a leaf table, Greenplum would compute statistics for all columns when merging statistics for the parent table, even if one or more columns had been configured with STATISTICS set to 0 (to disable statistics collection for that column). Greenplum no longer computes statistics for columns that have STATISTICS set to 0 when merging statistics for the parent table.

Release 6.16.2

Release Date: 2021-06-04

VMware Tanzu Greenplum 6.16.2 is a maintenance release that resolves several issues.

Resolved Issues

VMware Tanzu Greenplum 6.16.2 resolves these issues:

Cluster Management

178140424
Resolved an issue where, when gprecoverseg -v was invoked, pg_rewind's log was generated but was not preserved in case of an incremental recovery failure.

Server

178280660
Resolved an issue with the Postgres planner partition selection when the types of the partitioning key and the search values are different.
31436
Resolved an issue in which, when running \df+ on a function that has exec location INITPLAN, the Execute on column did not properly display "initplan".
31335
Resolved an out of memory condition that could occur because the server held certain data contexts longer than necessary.
12015
Fixed an inconsistency between master and segments regarding the value of collname when creating a DOMAIN.
11999
Resolved an issue in which CREATE MATERIALIZED VIEW was failing with ERROR: division by zero when WITH NO DATA was being specified.

Query Executor

31439
Resolved an issue which caused the database to PANIC due to a double free of the memory context TupleSort.

Release 6.16.1

Release Date: 2021-5-21

VMware Tanzu Greenplum 6.16.1 is a maintenance release that resolves several issues.

Resolved Issues

VMware Tanzu Greenplum 6.16.1 resolves these issues:

n/a - gprecoverseg
Resolved an issue where using gprecoverseg to perform an incremental recovery removed log files from the pg_log directory. gprecoverseg now retains files under pg_log so that they can be used for troubleshooting after an incremental recovery.
177336745 - Cluster Management
Resolved an issue with gpconfig where the process hung when cancelling it due to hosts being unreachable.
31453, 177970816 - Query Optimizer

Resolved an issue in which the optimizer was failing with a segfault while querying a view, if the view had a join between tables and any of the table had a column that was dropped after view creation.

31415 - Server
Resolved an issue in which REFRESH MATERIALIZED QUERY was failing if the WHERE clause included an embedded query.
31405 - Server
Resolved an issue in which certain queries resulted in a master panic because of the way temporary files were being tracked.
31383 - Server
Resolved an issue that was causing slow query performance for view creation when using the NOT in clause.
31380 - Cluster Management
The new functionality for gprecoverseg to ignore segments on unreachable hosts had introduced an issue for the gprecoverseg -p newhost option as it skipped the case of a new host replacing an unreachable host. This issue is now resolved.
31368 - Cluster Management
Resolved an issue with gpmovemirror which caused the wrong error message to be printed when there was an issue with the provided configuration file.
31361 - Query Executor
Resolved an issue in which multiple segments failed with PANIC errors due to how newly allocated memory was being handled.
31337 - Query Optimizer
Resolved an issue in which the optimizer was crashing – instead of simply throwing an error – when it was running out of memory.
31297 - Server
Resolved an issue in which VACUUM FULL ANALYZE was failing to clear some tables of bloat.
31279 - Server
Resolved an issue with incremental recovery when a failed segment that was acting as primary had to be recovered. The issue was caused by WAL files required by the recovery being removed.
31110 - Server
Resolved an issue in which a View with an unknown field could not be restored because columns were not being correctly cast from type cstring to type date.
30762 - Server
Resolved an issue with gpload in which a global transaction was not aborted when the session was reset.
29998 - Query Optimizer
Resolved an issue in which the optimizer was returning incorrect results when the query involved a CTE with an EXCEPT clause. The issue occurred because the query optimizer did not add scalar casts for any input columns whose types did not match the output types of the SetOp.

Release 6.16.0

Release Date: 2021-4-22

VMware Tanzu Greenplum 6.16.0 is a minor release that includes feature changes and resolves several issues.

Features

Greenplum Database 6.16.0 includes these new and changed features:

  • Greenplum Resource Groups include now a new mode for assigning CPU resources by percentage, Ceiling enforcement mode, apart from the existing Elastic mode.
  • The PXF version 6.0.0 distribution is available with this release; you can download it from the Release Download directory named Greenplum Platform Extension Framework on VMware Tanzu Network. Refer to the PXF documentation for information about this release and for installation and upgrade instructions.
  • Greenplum Streaming Server (GPSS) version 1.5.3 is included, which includes changes and bug fixes. Refer to the GPSS Documentation for more information about this release and for upgrade instructions.
  • Greenplum Database 6.16.0 includes MADlib version 1.18.0, which introduces new Deep Learning features, improvements, and bug fixes. See the MADlib 1.18.0 Release Notes for a complete list of changes.
  • The gp_sparse_vector module now installs its functions and objects into a schema named sparse_vector. See Resolved Issue 31360.
    Note: If you are using gp_sparse_vector in your current Greenplum Database distribution, review Upgrading the gp_sparse_vector Module for upgrade implications and instructions.
  • The default value of the optimizer_join_order server configuration parameter is changed from exhaustive2 to exhaustive, which was the default used in Greenplum versions prior to version 6.14.0. The Greenplum Database Query Optimizer (GPORCA) uses this configuration parameter to identify the join enumeration algorithm for a query. See Resolved Issue 31391.

Resolved Issues

VMware Tanzu Greenplum 6.16.0 resolves these issues:

31391 - Query Optimizer
Resolves an issue where a query using the GPORCA query optimizer took longer to complete on Greenplum Database version 6.15.0 with the optimizer_join_order server configuration parameter default value of exhaustive2. The default value of this configuration parameter is changed (back to) exhaustive.
31360 - gp_sparse_vector
Resolves an issue where the gp_sparse_vector array_agg() function overrode the system pg_catalog.array_agg() function and returned a different type of array. The gp_sparse_vector module now installs its functions and objects into a separate schema named sparse_vector.
31333 - Optimizer
GPORCA would generate invalid plans for certain queries with scalar subquery in the projection list, and a predicate that could use an underlying index. This could cause Greenplum Database to PANIC when running certain functions with the Optimizer enabled. This issue has now been resolved.
31308 - Server
Fixed an issue of queries hanging when logging is configured to write NOTICE level messages.
31296 - Server
Fixed an issue with gpcheckcat reporting an error when a VIEW is defined without a target and does not have an entry in pg_attribute.
31272 - Data Flow
In dual stack IPv4 and IPv6 hosts, gpfdist would bind to an IPv4 port but fail to bind to an IPv6 port, if there was another process listening on the same IPv6 port. This issue has been resolved.
31259 - Query Optimizer
When querying partitioned tables that had been defined with open boundaries, the Query Optimizer was performing unnecessary scans. This issue has been resolved.
31247 - Cluster Management
gpstate would log ERROR/FATAL messages relating to gpexapnd.status when no expansion was in progress, and when gpstate -s output showed a healthy cluster state. This issue has been resolved.
31346 - Query Optimizer
Resolved an issue with incorrect group pathkey that was causing the Greenplum database to go into recovery mode when using GROUP BY GROUPING SETS.
11655 - Query Optimizer
Fixed the error Lookup of object 0.0.0.0 in cache failed when querying a partitioned table.
11308 - Server

Fixes an issue where a bitmap index scan running concurrently with an index INSERT on a full bitmap page, occasionally failed to read the correct tid.

Upgrading from Greenplum 6.x to Greenplum 6.16

Note: Greenplum 6 does not support direct upgrades from Greenplum 4 or Greenplum 5 releases, or from earlier Greenplum 6 Beta releases.

See Upgrading from an Earlier Greenplum 6 Release to upgrade your existing Greenplum 6.x software to Greenplum 6.16.

Deprecated Features

Deprecated features will be removed in a future major release of Greenplum Database. VMware Tanzu Greenplum 6.x deprecates:

  • The gpsys1 utility.
  • The analzyedb option --skip_root_stats (deprecated since 6.2).

    If the option is specified, a warning is issued stating that the option will be ignored.

  • The server configuration parameter gp_statistics_use_fkeys (deprecated since 6.2).
  • The server configuration parameter gp_ignore_error_table (deprecated since 6.0).

    To avoid a Greenplum Database syntax error, set the value of this parameter to true when you run applications that execute CREATE EXTERNAL TABLE or COPY commands that include the now removed Greenplum Database 4.3.x INTO ERROR TABLE clause.

  • Specifying => as an operator name in the CREATE OPERATOR command (deprecated since 6.0).
  • The Greenplum external table C API (deprecated since 6.0).

    Any developers using this API are encouraged to use the new Foreign Data Wrapper API in its place.

  • Commas placed between a SUBPARTITION TEMPLATE clause and its corresponding SUBPARTITION BY clause, and between consecutive SUBPARTITION BY clauses in a CREATE TABLE command (deprecated since 6.0).

    Using this undocumented syntax will generate a deprecation warning message.

  • The timestamp format YYYYMMDDHH24MISS (deprecated since 6.0).

    This format could not be parsed unambiguously in previous Greenplum Database releases, and is not supported in PostgreSQL 9.4.

  • The createlang and droplang utilities (deprecated since 6.0).
  • The pg_resqueue_status system view (deprecated since 6.0).

    Use the gp_toolkit.gp_resqueue_status view instead.

  • The GLOBAL and LOCAL modifiers when creating a temporary table with the CREATE TABLE and CREATE TABLE AS commands (deprecated since 6.0).

    These keywords are present for SQL standard compatibility, but have no effect in Greenplum Database.

  • Using WITH OIDS or oids=TRUE to assign an OID system column when creating or altering a table (deprecated since 6.0).
  • Allowing superusers to specify the SQL_ASCII encoding regardless of the locale settings (deprecated since 6.0).

    This choice may result in misbehavior of character-string functions when data that is not encoding-compatible with the locale is stored in the database.

  • The @@@ text search operator (deprecated since 6.0).

    This operator is currently a synonym for the @@ operator.

  • The unparenthesized syntax for option lists in the VACUUM command (deprecated since 6.0).

    This syntax requires that the options to the command be specified in a specific order.

  • The plain pgbouncer authentication type (auth_type = plain) (deprecated since 4.x).

Migrating Data to Greenplum 6

Greenplum 6 supports direct upgrades, using gpupgrade, from Greenplum 5.x releases to Greenplum 6.x. For more information, see the gpupgrade documentation. See also Migrating Data from Greenplum 4.3 or 5 for guidelines and considerations for migrating existing Greenplum data to Greenplum 6, using standard backup and restore procedures.

Known Issues and Limitations

VMware Tanzu Greenplum 6 has these limitations:

  • Upgrading a Greenplum Database 4 release to VMware Tanzu Greenplum 6 is not supported. Upgrading a Greenplum Database 5.x release to VMware Tanzu Greenplum 6 is supported via gpupgrade. For more information, see the gpupgrade documentation.
  • MADlib, GPText, and PostGIS are not yet provided for installation on Ubuntu systems.
  • Greenplum for Kubernetes is not yet provided with this release.

The following table lists key known issues in VMware Tanzu Greenplum 6.x.

Table 1. Key Known Issues in VMware Tanzu Greenplum 6.x
Category Issue Description
31818 Server The fix for issue 11308, added in Greenplum 6.16, introduced a regression that could cause incorrect results for queries having a bitmap index condition. This problem is resolved in 6.17.7 by reverting the original fix.
Backup/Restore N/A Restoring the Greenplum Database backup for a table fails in Greenplum 6 versions earlier than version 6.10 when a replicated table has an inheritance relationship to/from another table that was assigned via an ALTER TABLE ... INHERIT statement after table creation.
Workaround: Use the following SQL commands to determine if Greenplum Database includes any replicated tables that inherit from a parent table, or if there are replicated tables that are inherited by a child table: 
SELECT inhrelid::regclass FROM pg_inherits,
  gp_distribution_policy dp
WHERE inhrelid=dp.localoid AND dp.policytype='r';
SELECT inhparent::regclass FROM pg_inherits,
  gp_distribution_policy dp
WHERE inhparent=dp.localoid AND dp.policytype='r';

If these queries return any tables, you may choose to run gprestore with the -–on-error-continue flag to not fail the entire restore when this issue is hit. Or, you can specify the list of tables returned by the queries to the -–exclude-table-file option to skip those tables during restore. You must recreate and repopulate the affected tables after restore.
  168548176 When using gpbackup to back up a Greenplum Database 5.7.1 or earlier 5.x release with resource groups enabled, gpbackup returns a column not found error for t6.value AS memoryauditor.
gpfdist 170824967 For Greenplum Database 6.x, a command that accesses an external table that uses the gpfdists protocol fails if the external table does not use an IP address when specifying a host system in the LOCATION clause of the external table definition.
Greenplum Client/Load Tools on Windows N/A The Greenplum Database client and load tools on Windows have not been tested with Active Directory Kerberos authentication.
Greenplum Installation N/A For Greenplum Database 6.16.1 and 6.16.2, the library libevent is not listed as a dependency for the rpm for RHEL 7. Resolved in 6.16.3.
Loading Data from External Tables 29703 Due to limitations in the Greenplum Database external table framework, Greenplum Database cannot log the following types of errors that it encounters while loading data:
  • data type parsing errors
  • unexpected value type errors
  • data type conversion errors
  • errors returned by native and user-defined functions
LOG ERRORS returns error information for data exceptions only. When it encounters a parsing error, Greenplum terminates the load job, but it cannot log and propagate the error back to the user via gp_read_error_log().

Workaround: Clean the input data before loading it into Greenplum Database.
Logging 30522 Greenplum Database may write a FATAL message to the standby master or mirror log stating that the database system is in recovery mode when the instance is synchronizing with the master and Greenplum attempts to contact it before the operation completes. Ignore these messages and use gpstate -f output to determine if the standby successfully synchronized with the Greenplum master; the command returns Sync state: sync if it is synchronized.
Materialized Views N/A By default, certain gp_toolkit views do not display data for materialized views. If you want to include this information in gp_toolkit view output, you must redefine a gp_toolkit internal view as described in Including Data for Materialized Views.
PL/R 164791118 PL/R cannot be installed using the deprecated createlang utility, and displays the error:
createlang: language installation failed: ERROR:
no schema has been selected to create in
Workaround: Use CREATE EXTENSION to install PL/R, as described in the documentation.
Postgres Planner 30537 The Postgres Planner generates a very large query plan that causes out of memory issues for the following type of CTE (common table expression) query: the WITH clause of the CTE contains a partitioned table with a large number partitions, and the WITH reference is used in a subquery that joins another partitioned table.

Workaround: If possible, use the GPORCA query optimizer. With the server configuration parameter optimizer=on, Greenplum Database attempts to use GPORCA for query planning and optimization when possible and falls back to the Postgres Planner when GPORCA cannot be used. Also, the specified type of query might require a long time to complete.
PXF N/A Starting in 6.x, Greenplum does not bundle cURL and instead loads the system-provided library. PXF requires cURL version 7.29.0 or newer. The officially-supported cURL for the CentOS 6.x and Red Hat Enterprise Linux 6.x operating systems is version 7.19.*. Greenplum Database 6 does not support running PXF on CentOS 6.x or RHEL 6.x due to this limitation.

Workaround: Upgrade the operating system of your Greenplum Database 6 hosts to CentOS 7+ or RHEL 7+, which provides a cURL version suitable to run PXF.
  168957894 The PXF Hive Connector does not support using the Hive* profiles to access Hive transactional tables.

Workaround: Use the PXF JDBC Connector to access Hive.
Query Optimizer 31010 A view created in Greenplum Database 5.28.3 or older that specified an external table in the FROM clause, and that was migrated to Greenplum Database 6.x, always falls back to the Postgres Planner when queried.

Workaround: If you migrated a view from Greenplum Database 5.28.3 or earlier, and the view specified an external table in the FROM clause, you must drop and recreate the view in Greenplum 6.x to ensure that the Query Optimizer is exercised when you query the view.
Resource Management 30594 Resource queue-related statistics may be inaccurate in certain cases. VMware recommends that you use the resource group resource management scheme that is available in Greenplum 6.
Spark Connector N/A This version of Greenplum is not compatible with Greenplum-Spark Connector versions earlier than version 1.7.0, due to a change in how Greenplum handles distributed transaction IDs.

Differences Compared to Open Source Greenplum Database

VMware Tanzu Greenplum 6.x includes all of the functionality in the open source Greenplum Database project and adds:
  • Product packaging and installation script
  • Support for QuickLZ compression. QuickLZ compression is not provided in the open source version of Greenplum Database due to licensing restrictions.
  • Support for data connectors:
    • Greenplum-Spark Connector
    • Greenplum-Informatica Connector
    • Greenplum-Kafka Integration
    • Greenplum Streaming Server
  • Data Direct ODBC/JDBC Drivers
  • Greenplum PostGIS Extension
  • gpcopy utility for copying or migrating objects between Greenplum systems
  • Support for managing Greenplum Database using VMware Tanzu Greenplum Command Center
  • Support for full text search and text analysis using VMware Tanzu GPText
  • Greenplum backup plugin for DD Boost
  • Backup/restore storage plugin API