Configuring PXF Hadoop Connectors (Optional)
A newer version of this documentation is available. Click here to view the most up-to-date release of the Greenplum 5.x documentation.
PXF is compatible with Cloudera, Hortonworks Data Platform, MapR, and generic Apache Hadoop distributions. This topic describes how configure the PXF Hadoop, Hive, and HBase connectors.
If you do not want to use the Hadoop-related PXF connectors, then you do not need to perform this procedure.
Configuring PXF Hadoop connectors involves copying configuration files from your Hadoop cluster to the Greenplum Database master host. If you are using the MapR Hadoop distribution, you must also copy certain JAR files to the master host. Before you configure the PXF Hadoop connectors, ensure that you can copy files from hosts in your Hadoop cluster to the Greenplum Database master.
In this procedure, you copy Hadoop configuration files to the
$PXF_CONF/servers/default directory on the Greenplum Database master host. You may also copy libraries to
$PXF_CONF/lib for MapR support. You then synchronize the PXF configuration on the master host to the segment hosts. (PXF creates the
$PXF_CONF/* directories when you run
pxf cluster init.)
Note: After you complete the configuration procedure, you will have configured the PXF default Hadoop server. End users need not provide a
SERVER option in a
CREATE EXTERNAL TABLE command when they access the default Hadoop server configuration.
Perform the following procedure to configure the desired PXF Hadoop-related connectors on the Greenplum Database master host. After you configure the connectors, you will use the
pxf cluster sync command to copy the PXF configuration to the segment hosts in your Greenplum Database cluster.
Log in to your Greenplum Database master node:
$ ssh gpadmin@<gpmaster>
PXF requires information from
core-site.xmland other Hadoop configuration files. Copy the
yarn-site.xmlHadoop configuration files from your Hadoop cluster NameNode host to the current host using your tool of choice. (Your file paths may differ based on the Hadoop distribution in use.) For example, these commands use
scpto copy the files:
gpadmin@gpmaster$ cd $PXF_CONF/servers/default gpadmin@gpmaster$ scp hdfsuser@namenode:/etc/hadoop/conf/core-site.xml . gpadmin@gpmaster$ scp hdfsuser@namenode:/etc/hadoop/conf/hdfs-site.xml . gpadmin@gpmaster$ scp hdfsuser@namenode:/etc/hadoop/conf/mapred-site.xml . gpadmin@gpmaster$ scp hdfsuser@namenode:/etc/hadoop/conf/yarn-site.xml .
If you plan to use the PXF Hive connector to access Hive table data, similarly copy the Hive configuration to the Greenplum Database master host. For example:
gpadmin@gpmaster$ scp hiveuser@hivehost:/etc/hive/conf/hive-site.xml .
If you plan to use the PXF HBase connector to access HBase table data, similarly copy the HBase configuration to the Greenplum Database master host. For example:
gpadmin@gpmaster$ scp hbaseuser@hbasehost:/etc/hbase/conf/hbase-site.xml .
If you are using PXF with the MapR Hadoop distribution, you must copy certain JAR files from your MapR cluster to the Greenplum Database master host. (Your file paths may differ based on the version of MapR in use.) For example, these commands use
scpto copy the files:
gpadmin@gpmaster$ cd $PXF_CONF/lib gpadmin@gpmaster$ scp mapruser@maprhost:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/maprfs-5.2.2-mapr.jar . gpadmin@gpmaster$ scp mapruser@maprhost:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/lib/hadoop-auth-2.7.0-mapr-1707.jar . gpadmin@gpmaster$ scp mapruser@maprhost:/opt/mapr/hadoop/hadoop-2.7.0/share/hadoop/common/hadoop-common-2.7.0-mapr-1707.jar .
Synchronize the PXF configuration to each Greenplum Database segment host. For example:
gpadmin@gpmaster$ $GPHOME/pxf/bin/pxf cluster sync
PXF accesses Hadoop services on behalf of Greenplum Database end users. By default, PXF tries to access HDFS, Hive, and HBase using the identity of the Greenplum Database user account that logs into Greenplum Database. In order to support this functionality, you must configure proxy settings for Hadoop, as well as for Hive and HBase if you intend to use those PXF connectors. Follow procedures in Configuring User Impersonation and Proxying to configure user impersonation and proxying for Hadoop services, or to turn off PXF user impersonation.
Grant read permission to the HDFS files and directories that will be accessed as external tables in Greenplum Database. If user impersonation is enabled (the default), you must grant this permission to each Greenplum Database user/role name that will use external tables that reference the HDFS files. If user impersonation is not enabled, you must grant this permission to the
If your Hadoop cluster is secured with Kerberos, you must configure PXF and generate Kerberos principals and keytabs for each segment host as described in Configuring PXF for Secure HDFS.
If you update your Hadoop, Hive, or HBase configuration while the PXF service is running, you must re-sync the PXF configuration to your Greenplum Database cluster and restart PXF on each segment host in the cluster. For example:
gpadmin@gpmaster$ $GPHOME/pxf/bin/pxf cluster sync gpadmin@gpmaster$ $GPHOME/pxf/bin/pxf cluster stop gpadmin@gpmaster$ $GPHOME/pxf/bin/pxf cluster start