Tanzu Greenplum Platform Extension Framework 5.x Release Notes

The Tanzu Greenplum Platform Extension Framework (PXF) is included in the Tanzu Greenplum distribution in Greenplum versions including and older than 5.28 and 6.11. PXF for Redhat/CentOS and Oracle Enterprise Linux is updated and distributed independently of Greenplum Database starting with PXF version 5.13.0.

You may need to download and install the PXF package to obtain the most recent version of this component.

Supported Platforms

The independent PXF distribution is compatible with these operating system platform and Greenplum versions:

PXF is compatible with these Java and Hadoop component versions:

Release 5.16.4

Release Date: September 1, 2021

Resolved Issues

PXF 5.16.4 resolves this issue:

Release 5.16.3

Release Date: August 20, 2021

Changes

PXF introduces a new property to the pxf-site.xml per-server configuration file to resolve issue 31657. PXF uses this property, pxf.sasl.connection.retries, to specify the maximum number of times that it retries a SASL connection request to an external data source after a refused connection returns a GSS initiate failed error.

Resolved Issues

PXF 5.16.3 resolves this issue:

Additional Information about Resolved Issue 31657

You can determine if this issue is the cause of your external table error when accessing Hadoop as follows:

1. Verify that you are accessing a Kerberos-secured Hadooop environment.

2. Examine the error message returned by the client and confirm that it contains this text:

   org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
   

3. Examine the catalina.<date>.log on the segment host that reported the error and confirm that it contains this error message:

   <date> com.sun.jersey.spi.container.ContainerResponse mapMappableContainerException
   SEVERE: The exception contained within MappableContainerException could not be mapped to a response, re-throwing to the HTTP container
   org.apache.hadoop.ipc.RemoteException(javax.security.sasl.SaslException): GSS initiate failed
   

4. Inspect the Hadoop NameNode log(s) and confirm that it contains this message:

   <date> INFO org.apache.hadoop.ipc.Server: Socket Reader #1 for port 8020: readAndProcess from client <address> threw exception [javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism level: Request is a replay (34))]]
   

If you can confirm each point above, this issue is indeed the cause of your error. You must upgrade to PXF 5.16.3, and optionally perform certain configuration tasks:

1. (Required) Upgrade to PXF version 5.16.3.

2. (Recommended) Configure PXF to use a host-specific Kerberos principal for each segment host. If you specify the following pxf.service.kerberos.principal property setting in the PXF server’s pxf-site.xml file, PXF automatically replaces _HOST with the FQDN of the segment host:

   <property>
     <name>pxf.service.kerberos.principal</name>
     <value>gpadmin/_HOST@REALM.COM</value>
   </property>
   

   (If you update pxf-site.xml, be sure to pxf cluster sync to propagate the changes to the Greenplum Database standby and all segment hosts in the cluster.)

Release 5.16.2

Release Date: February 25, 2021

Resolved Issues

PXF 5.16.2 resolves these issues:

Release 5.16.1

Release Date: February 5, 2021

Resolved Issues

PXF 5.16.1 resolves this issue:

Release 5.16

Release Date: November 6, 2020

New and Changed Features

PXF 5.16.0 includes these new and changed features:

• Version 5.16 is the first standalone PXF release that includes a .deb installation package for Ubuntu 18.04 LTS systems.
• PXF adds support for reading files from, and writing files to, a network file system mounted on each Greenplum Database segment host. See Accessing Files on a Network File System with PXF for prerequisites, configuration, and usage information for this feature.
• PXF disallows specifying relative paths and environment variables in the CREATE EXTERNAL TABLE LOCATION clause file path.
• PXF adds the new pxf.fs.basePath property to the PXF pxf-site.xml template file. The property is commented out by default; you set this property to specify the base path from which PXF accesses the path that you specify in the CREATE EXTERNAL TABLE LOCATION clause. See About the pxf.fs.basePath Property for more information.
• The pxf.service.user.name property in the PXF pxf-site.xml template file is now commented out by default, and the file now includes an enhanced description of the property. Additionally, the documentation has been enhanced to provide specific use cases and configuration scenarios for access to secured and non-secured Hadoop clusters. See secured cluster use cases and non-secured cluster use cases.
• The default value of the jdbc.pool.property.maximumPoolSize property in the jdbc-site.xml template file was increased from 5 to 15 to better support out-of-the-box reads and writes of large amounts of data.
• PXF now supports reading from, and writing to, external tables altered by dropping columns.
• PXF adds the new --skip-register flag to the pxf [cluster] init command. This flag instructs PXF to skip the initialization step that copies the PXF extension files to the Greenplum installation on the host(s).
• The PXF Hive profile now supports column projection and predicate pushdown when you use it to access Hive tables STORED AS Parquet.
• Column projection and predicate pushdown are now enabled by default when you use PXF and the Hive profile to access Hive tables STORED AS Parquet, RCFile, and ORC.
• PXF now supports using the Hive profile to read data from a Hive table STORED AS Parquet when the underlying Parquet file(s) has a different column order than the defining Hive table.
• PXF now reduces and optimizes its memory usage during fragmentation for Hive* profiles.

Resolved Issues

PXF 5.16 resolves these issues:

Release 5.15.1

Release Date: September 11, 2020

Changes

PXF 5.15.1 includes these changes:

• PXF bundles a new version of Tomcat, 7.0.105.
• PXF improves the performance of Parquet write operations (see Resolved Issue 30788, 30779) by:
  • No longer splitting files that are over 128MB in size.
  • Bundling Parquet version 1.11.1 libraries.
  • Providing a new ENABLE_DICTIONARY option to enable/disable dictionary encoding when PXF writes Parquet data.
  • Using the Parquet logical date and physical int32 types when writing dates. See Resolved Issue 174433819.
• When dictionary-encoding is enabled, the default DICTIONARY_PAGE_SIZE that PXF uses when writing Parquet data is now 1 * 1024 * 1024 (it was previously 1 * 512 * 1024).
• PXF provides integrated native library registration support by exposing the new user configuration directory $PXF_CONF/lib/native and a template for setting the LD_LIBRARY_PATH option. See Resolved Issue 264 and Registering PXF Library Dependencies.

Resolved Issues

PXF 5.15.1 resolves these issues:

Release 5.15.0

Release Date: August 25, 2020

New and Changed Features

PXF 5.15.0 includes these new and changed features:

• PXF bundles the opencsv library to satisfy a missing transitive dependency that is required when PXF reads Hive tables created with the OpenCSVSerde serialization/deserialization class.
• PXF bundles newer jackson-core, jackson-databind, and jackson-annotations supporting libraries.
• PXF supports bzip2 and xz compression when reading from or writing to Avro files.
• PXF introduces a new option named SKIP_HEADER_COUNT=<N> that you can use to instruct PXF to skip the first N lines in the first split of a text file.
• PXF includes improvements to Hive error handling and error surfacing.
• PXF no longer restricts operations using bzip2 compression to a single thread.
• PXF 5.15.0 deprecates and ignores the THREAD-SAFE custom option setting. All query and write operations on a PXF external table are now always thread-safe.

Resolved Issues

PXF 5.15.0 resolves these issues:

Release 5.14.0

Release Date: July 7, 2020

New and Changed Features

PXF 5.14.0 includes these new and changed features:

• PXF supports the deflate and snappy compression codecs when writing Avro data to an external data store. By default, PXF now compresses all Avro data with the deflate codec before writing it to the external store.
• Before writing Avro data, PXF converts smallint-type columns to the int data type. You must specify an int-type column in an external table definition to read this data.

Resolved Issues

PXF 5.14.0 resolves these issues:

Release 5.13.0

Release Date: June 30, 2020

New and Changed Features

PXF 5.13.0 includes these new and changed features since PXF 5.12.0:

• PXF 5.13.0 is the first standalone release of PXF for RedHat/CentOS that is distributed separately from Greenplum Database.

Resolved Issues

PXF 5.13.0 resolves these issues:

Deprecated Features

Deprecated features may be removed in a future major release of PXF. PXF version 5.x deprecates:

• The THREAD-SAFE custom option setting. All query and write operations are thread-safe (deprecated since PXF version 5.15.0).
• The PXF_USER_IMPERSONATION, PXF_PRINCIPAL, and PXF_KEYTAB settings in the pxf-env.sh file. You can use the pxf-site.xml file to configure Kerberos and impersonation settings for your new Hadoop server configurations (deprecated since PXF version 5.10.0).
• The pxf.impersonation.jdbc property setting in the jdbc-site.xml file. You can use the pxf.service.user.impersonation property to configure user impersonation for a new JDBC server configuration (deprecated since PXF version 5.10.0).
• The HDFS profile names for the Text, Avro, JSON, Parquet, and SequenceFile data formats (deprecated since PXF version 5.0.1). Refer to Connectors, Data Formats, and Profiles in the PXF Hadoop documentation for more information.

Known Issues and Limitations

PXF 5.x has these known issues and limitations: