Configuring Your Systems and Installing Greenplum

Configuring Your Systems and Installing Greenplum

This chapter describes how to prepare your operating system environment for Greenplum, and install the Greenplum Database software binaries on all of the hosts that will comprise your Greenplum Database system. Perform the following tasks in order:

  1. Make sure your systems meet the System Requirements
  2. Setting the Greenplum Recommended OS Parameters
  3. (master only) Running the Greenplum Installer
  4. Installing and Configuring Greenplum on all Hosts
  5. (Optional) Installing Oracle Compatibility Functions
  6. (Optional) Installing Greenplum Database Extensions
  7. Creating the Data Storage Areas
  8. Synchronizing System Clocks
  9. Next Steps

Unless noted, these tasks should be performed for all hosts in your Greenplum Database array (master, standby master and segments).

Note: For information about upgrading Greenplum Database from a previous version, see the Greenplum Database Release Notes for the release that you are installing.

System Requirements

The following table lists minimum recommended specifications for servers intended to support Greenplum Database in a production environment. Greenplum also provides hardware build guides for its certified hardware platforms. It is recommended that you work with a Greenplum Systems Engineer to review your anticipated environment to ensure an appropriate hardware configuration for Greenplum Database.

Table 1. System Prerequisites for Greenplum Database 4.3
Operating System SUSE Linux Enterprise Server 11 SP2

CentOS 5.0 or higher

RedHat Enterprise Linux 5.0 or higher

Oracle Unbreakable Linux 5.5

Note: See the Greenplum Database Release Notes for current supported platform information.
File Systems
  • xfs required for data storage on SUSE Linux and Red Hat (ext3 supported for root file system)
Minimum CPU Pentium Pro compatible (P3/Athlon and above)
Minimum Memory 16 GB RAM per server
Disk Requirements
  • 150MB per host for Greenplum installation
  • Approximately 300MB per segment instance for meta data
  • Appropriate free space for data with disks at no more than 70% capacity
  • High-speed, local storage
Network Requirements Gigabit Ethernet within the array

Dedicated, non-blocking switch

Software and Utilities bash shell

GNU tar

GNU zip

GNU ed (used by the Greenplum Database installer)

Important: For all Greenplum Database host systems, the SELinux must be disabled. Pivotal recommends that firewall software such as iptables also be disabled. You can enable iptables if it is required for security purposes. For information about enabling and configuring iptables, see Enabling iptables.

For example, this command checks the status of iptables when run as root:

# /sbin/chkconfig --list iptables

This is the output if iptables is disabled.

iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

This command checks the status of SELinux when run as root:

# sestatus
SELinuxstatus: disabled

One method of disabling iptables is to become root, run this command, and then reboot the system:

/sbin/chkconfig iptables off

You can disable SELinux by editing the /etc/selinux/config file. As root, change the value of the SELINUX parameter in the config file and reboot the system:


For information about disabling firewall software, see the documentation for the firewall or your operating system. For information about disabling SELinux, see the SELinux documentation.

Important: SSL is supported only on the Greenplum Database master host system. It is not supported on the segment host systems.

Setting the Greenplum Recommended OS Parameters

Greenplum requires the certain operating system (OS) parameters be set on all hosts in your Greenplum Database system (masters and segments).

In general, the following categories of system parameters need to be altered:

  • Shared Memory - A Greenplum Database instance will not work unless the shared memory segment for your kernel is properly sized. Most default OS installations have the shared memory values set too low for Greenplum Database. On Linux systems, you must also disable the OOM (out of memory) killer.
  • Network - On high-volume Greenplum Database systems, certain network-related tuning parameters must be set to optimize network connections made by the Greenplum interconnect.
  • User Limits - User limits control the resources available to processes started by a user's shell. Greenplum Database requires a higher limit on the allowed number of file descriptors that a single process can have open. The default settings may cause some Greenplum Database queries to fail because they will run out of file descriptors needed to process the query.

Linux System Settings

  • Set the following parameters in the /etc/sysctl.conf file and reboot:
    kernel.shmmax = 500000000
    kernel.shmmni = 4096
    kernel.shmall = 4000000000
    kernel.sem = 250 512000 100 2048
    kernel.sysrq = 1
    kernel.core_uses_pid = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.msgmni = 2048
    net.ipv4.tcp_syncookies = 1
    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_max_syn_backlog = 4096
    net.ipv4.conf.all.arp_filter = 1
    net.ipv4.ip_local_port_range = 1025 65535
    net.core.netdev_max_backlog = 10000
    net.core.rmem_max = 2097152
    net.core.wmem_max = 2097152
    vm.overcommit_memory = 2
  • Set the following parameters in the /etc/security/limits.conf file:
    * soft nofile 65536
    * hard nofile 65536
    * soft nproc 131072
    * hard nproc 131072

    For RedHat Enterprise Linux 6.x and CentOS 6.x, parameter values in the /etc/security/limits.d/90-nproc.conf file override the values in the limits.conf file. If a parameter value is set in both conf files, ensure that the parameter is set properly in the 90-nproc.conf file. The Linux module pam_limits sets user limits by reading the values from the limits.conf file and then from the 90-nproc.conf file. For information about PAM and user limits, see the documentation on PAM and pam_limits.

  • XFS is the preferred file system on Linux platforms for data storage. Greenplum recommends the following XFS mount options:

    See the manual page (man) for the mount command for more information about using that command (man mount opens the man page).

    The XFS options can also be set in the /etc/fstab file. This example entry from an fstab file specifies the XFS options.

    /dev/data /data xfs nodev,noatime,inode64,allocsize=16m 0 0
  • The Linux disk I/O scheduler for disk access supports different policies, such as CFQ, AS, and deadline.

    Greenplum recommends the following scheduler option: deadline. To specify a scheduler until the next system reboot, run the following:

    # echo schedulername > /sys/block/devname/queue/scheduler

    For example:

    # echo deadline > /sys/block/sbd/queue/scheduler

    You can specify the I/O scheduler at boot time with the elevator kernel parameter. Add the parameter elevator=deadline to the kernel command in the file /boot/grub/grub.conf, the GRUB boot loader configuration file. After adding the parameter, reboot the system.

  • Each disk device file should have a read-ahead (blockdev) value of 16384.

    To verify the read-ahead value of a disk device:

    # /sbin/blockdev --getra devname

    For example:

    # /sbin/blockdev --getra /dev/sdb

    To set blockdev (read-ahead) on a device:

    # /sbin/blockdev --setra bytes devname

    For example:

    # /sbin/blockdev --setra 16384 /dev/sdb

    See the manual page (man) for the blockdev command for more information about using that command (man blockdev opens the man page).

  • Edit the /etc/hosts file and make sure that it includes the host names and all interface address names for every machine participating in your Greenplum Database system.
  • Disable Transparent Huge Pages (THP). RedHat Enterprise Linux 6.0 or higher enables THP by default. THP degrades Greenplum Database performance. One way to disable THP is by disabling the setting in the /sys/kernel file system. This example echo command disables THP:
    $ echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled

    Place the echo command in /etc/rc.local to ensure THP is disabled on each reboot.

    This cat command checks the state of THP. The output indicates that THP is disabled.

    $ cat /sys/kernel/mm/redhat_transparent_hugepage/enabled
    always [never]

    For more information about Transparent Huge Pages, see your operating system documentation.

Mac OS X System Settings

  • Add the following to /etc/sysctl.conf:
  • Add the following line to /etc/hostconfig:

Running the Greenplum Installer

To configure your systems for Greenplum Database, you will need certain utilities found in $GPHOME/bin of your installation. Log in as root and run the Greenplum installer on the machine that will be your master host.

Important: If you are upgrading Greenplum Database, uninstall the Greenplum Database the gNet extension package if it is installed.

The gNet extension package contains the software for the gphdfs protocol. For Greenplum Database 4.3.1 and later releases, the extension is bundled with Greenplum Database. The files for gphdfs are installed in $GPHOME/lib/hadoop.

To install the Greenplum binaries on the master host

  1. Download or copy the installer file to the machine that will be the Greenplum Database master host. Installer files are available from Greenplum for RedHat (32-bit and 64-bit), and SuSe Linux 64-bit platforms.
  2. Unzip the installer file where PLATFORM is either RHEL5-i386 (RedHat 32-bit), RHEL5-x86_64 (RedHat 64-bit), or SuSE10-x86_64 (SuSe Linux 64 bit). For example:
    # unzip
  3. Launch the installer using bash. For example:
    # /bin/bash greenplum-db-4.3.x.x-PLATFORM.bin
  4. The installer will prompt you to accept the Greenplum Database license agreement. Type yes to accept the license agreement.
  5. The installer will prompt you to provide an installation path. Press ENTER to accept the default install path (/usr/local/greenplum-db-4.3.x.x), or enter an absolute path to an install location. You must have write permissions to the location you specify.
  6. Optional. The installer will prompt you to provide the path to a previous installation of Greenplum Database. For example: /usr/local/greenplum-db-4.3.x.x)

    This installation step will migrate any Greenplum Database add-on modules (postgis, pgcrypto, etc.) from the previous installation path to the path to the version currently being installed. This step is optional and can be performed manually at any point after the installation using the gpkg utility with the -migrate option. See gppkg for details.

    Press ENTER to skip this step.

  7. The installer will install the Greenplum software and create a greenplum-db symbolic link one directory level above your version-specific Greenplum installation directory. The symbolic link is used to facilitate patch maintenance and upgrades between versions. The installed location is referred to as $GPHOME.
  8. To perform additional required system configuration tasks and to install Greenplum Database on other hosts, go to the next task Installing and Configuring Greenplum on all Hosts.

About Your Greenplum Database Installation

  • — This file contains the environment variables for Greenplum Database. See Setting Greenplum Environment Variables.
  • GPDB-LICENSE.txt — Greenplum license agreement.
  • bin — This directory contains the Greenplum Database management utilities. This directory also contains the PostgreSQL client and server programs, most of which are also used in Greenplum Database.
  • demo — This directory contains the Greenplum demonstration programs.
  • docs — The Greenplum Database documentation (PDF files).
  • etc — Sample configuration file for OpenSSL.
  • ext — Bundled programs (such as Python) used by some Greenplum Database utilities.
  • include — The C header files for Greenplum Database.
  • lib — Greenplum Database and PostgreSQL library files.
  • sbin — Supporting/Internal scripts and programs.
  • share — Shared files for Greenplum Database.

Installing and Configuring Greenplum on all Hosts

When run as root, gpseginstall copies the Greenplum Database installation from the current host and installs it on a list of specified hosts, creates the Greenplum system user (gpadmin), sets the system user's password (default is changeme), sets the ownership of the Greenplum Database installation directory, and exchanges ssh keys between all specified host address names (both as root and as the specified system user).

About gpadmin

When a Greenplum Database system is first initialized, the system contains one predefined superuser role (also referred to as the system user), gpadmin. This is the user who owns and administers the Greenplum Database.

Note: If you are setting up a single node system, you can still use gpseginstall to perform the required system configuration tasks on the current host. In this case, the hostfile_exkeys would just have the current host name only.

To install and configure Greenplum Database on all specified hosts

  1. Log in to the master host as root:
    $ su -
  2. Source the path file from your master host's Greenplum Database installation directory:
    # source /usr/local/greenplum-db/
  3. Create a file called hostfile_exkeys that has the machine configured host names and host addresses (interface names) for each host in your Greenplum system (master, standby master and segments). Make sure there are no blank lines or extra spaces. For example, if you have a master, standby master and three segments with two network interfaces per host, your file would look something like this:

    Check the /etc/hosts file on your systems for the correct host names to use for your environment.

    The Greenplum Database segment host naming convention is sdwN where sdw is a prefix and N is an integer. For example, on a Greenplum Database DCA system, segment host names would be sdw1, sdw2 and so on. For hosts with multiple interfaces, the convention is to append a dash (-) and number to the host name. For example, sdw1-1 and sdw1-2 are the two interface names for host sdw1.

  4. Run the gpseginstall utility referencing the hostfile_exkeys file you just created. Use the -u and -p options to create the Greenplum system user (gpadmin) on all hosts and set the password for that user on all hosts. For example:
    # gpseginstall -f hostfile_exkeys -u gpadmin -p P@$$word

    Recommended security best practices:

  • Do not use the default password option for production environments.
  • Change the password immediately after installation.

Confirming Your Installation

To make sure the Greenplum software was installed and configured correctly, run the following confirmation steps from your Greenplum master host. If necessary, correct any problems before continuing on to the next task.

  1. Log in to the master host as gpadmin:
    $ su - gpadmin
  2. Source the path file from Greenplum Database installation directory:
    # source /usr/local/greenplum-db/
  3. Use the gpssh utility to see if you can login to all hosts without a password prompt, and to confirm that the Greenplum software was installed on all hosts. Use the hostfile_exkeys file you used for installation. For example:
    $ gpssh -f hostfile_exkeys -e ls -l $GPHOME

    If the installation was successful, you should be able to log in to all hosts without a password prompt. All hosts should show that they have the same contents in their installation directories, and that the directories are owned by the gpadmin user.

    If you are prompted for a password, run the following command to redo the ssh key exchange:

    $ gpssh-exkeys -f hostfile_exkeys

Installing Oracle Compatibility Functions

Optional. Many Oracle Compatibility SQL functions are available in Greenplum Database. These functions target PostgreSQL.

Before using any Oracle Compatibility Functions, you need to run the installation script $GPHOME/share/postgresql/contrib/orafunc.sql once for each database. For example, to install the functions in database testdb, use the command

$ psql –d testdb –f $GPHOME/share/postgresql/contrib/orafunc.sql

To uninstall Oracle Compatibility Functions, use the script:

Note: The following functions are available by default and can be accessed without running the Oracle Compatibility installer: sinh, tanh, cosh and decode.

For more information about Greenplum's Oracle compatibility functions, see "Oracle Compatibility Functions" in the Greenplum Database Utility Guide.

Installing Greenplum Database Extensions

Optional. Use the Greenplum package manager (gppkg) to install Greenplum Database extensions such as pgcrypto, PL/R, PL/Java, PL/Perl, and PostGIS, along with their dependencies, across an entire cluster. The package manager also integrates with existing scripts so that any packages are automatically installed on any new hosts introduced into the system following cluster expansion or segment host recovery.

See gppkg for more information, including usage.

Creating the Data Storage Areas

Every Greenplum Database master and segment instance has a designated storage area on disk that is called the data directory location. This is the file system location where the directories that store segment instance data will be created. The master host needs a data storage location for the master data directory. Each segment host needs a data directory storage location for its primary segments, and another for its mirror segments.

To create the data directory location on the master

The data directory location on the master is different than those on the segments. The master does not store any user data, only the system catalog tables and system metadata are stored on the master instance, therefore you do not need to designate as much storage space as on the segments.

  1. Create or choose a directory that will serve as your master data storage area. This directory should have sufficient disk space for your data and be owned by the gpadmin user and group. For example, run the following commands as root:
    # mkdir /data/master
  2. Change ownership of this directory to the gpadmin user. For example:
    # chown gpadmin /data/master
  3. Using gpssh, create the master data directory location on your standby master as well. For example:
    # source /usr/local/greenplum-db-4.3.x.x/ 
    # gpssh -h smdw -e 'mkdir /data/master'
    # gpssh -h smdw -e 'chown gpadmin /data/master'

To create the data directory locations on all segment hosts

  1. On the master host, log in as root:
    # su
  2. Create a file called hostfile_gpssh_segonly. This file should have only one machine configured host name for each segment host. For example, if you have three segment hosts:
  3. Using gpssh, create the primary and mirror data directory locations on all segment hosts at once using the hostfile_gpssh_segonly file you just created. For example:
    # source /usr/local/greenplum-db-4.3.x.x/ 
    # gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/primary'
    # gpssh -f hostfile_gpssh_segonly -e 'mkdir /data/mirror'
    # gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/primary'
    # gpssh -f hostfile_gpssh_segonly -e 'chown gpadmin /data/mirror'

Synchronizing System Clocks

Greenplum recommends using NTP (Network Time Protocol) to synchronize the system clocks on all hosts that comprise your Greenplum Database system. See for more information about NTP.

NTP on the segment hosts should be configured to use the master host as the primary time source, and the standby master as the secondary time source. On the master and standby master hosts, configure NTP to point to your preferred time server.

To configure NTP

  1. On the master host, log in as root and edit the /etc/ntp.conf file. Set the server parameter to point to your data center's NTP time server. For example (if was the IP address of your data center's NTP server):
  2. On each segment host, log in as root and edit the /etc/ntp.conf file. Set the first server parameter to point to the master host, and the second server parameter to point to the standby master host. For example:
    server mdw prefer
    server smdw
  3. On the standby master host, log in as root and edit the /etc/ntp.conf file. Set the first server parameter to point to the primary master host, and the second server parameter to point to your data center's NTP time server. For example:
    server mdw prefer
  4. On the master host, use the NTP daemon synchronize the system clocks on all Greenplum hosts. For example, using gpssh:
    # gpssh -f hostfile_gpssh_allhosts -v -e 'ntpd'

Enabling iptables

On Linux systems, you can configure and enable the iptables firewall to work with Greenplum Database.

Note: Greenplum Database performance might be impacted when iptables is enabled. You should test the performance of your application with iptables enabled to ensure that performance is acceptable.

For more information about iptables see the iptables and firewall documentation for your operating system.

How to Enable iptables

  1. As gpadmin, the Greenplum Database administrator, run this command on the Greenplum Database master host to stop Greenplum Database:
    $ gpstop -a
  2. On the Greenplum Database hosts:
    1. Update the file /etc/sysconfig/iptables based on the Example iptables Rules.
    2. As root user, run these commands to enable iptables:
      # chkconfig iptables on
      # service iptables start
  3. As gpadmin, run this command on the Greenplum Database master host to start Greenplum Database:
    $ gpstart -a
Warning: After enabling iptables, this error in the /var/log/messages file indicates that the setting for the iptables table is too low and needs to be increased.
ip_conntrack: table full, dropping packet.

As root user, run this command to view the iptables table value:

# sysctl net.ipv4.netfilter.ip_conntrack_max

The following is the recommended setting to ensure that the Greenplum Database workload does not overflow the iptables table. The value might need to be adjusted for your hosts: net.ipv4.netfilter.ip_conntrack_max=6553600

You can update /etc/sysctl.conf file with the value. For setting values in the file, see Setting the Greenplum Recommended OS Parameters.

To set the value until the next reboots run this command as root.

# sysctl net.ipv4.netfilter.ip_conntrack_max=6553600

Example iptables Rules

When iptables is enabled, iptables manages the IP communication on the host system based on configuration settings (rules). The example rules are used to configure iptables for Greenplum Database master host, standby master host, and segment hosts.

The two sets of rules account for the different types of communication Greenplum Database expects on the master (primary and standby) and segment hosts. The rules should be added to the /etc/sysconfig/iptables file of the Greenplum Database hosts. For Greenplum Database, iptables rules should allow the following communication:

  • For customer facing communication with the Greenplum Database master, allow at least postgres and 28080 (eth1 interface in the example).
  • For Greenplum Database system interconnect, allow communication using tcp, udp, and icmp protocols (eth4 and eth5 interfaces in the example).

    The network interfaces that you specify in the iptables settings are the interfaces for the Greenplum Database hosts that you list in the hostfile_gpinitsystem file. You specify the file when you run the gpinitsystem command to intialize a Greenplum Database system. See Initializing a Greenplum Database System for information about the hostfile_gpinitsystem file and the gpinitsystem command.

  • For the administration network on a Greenplum DCA, allow communication using ssh, snmp, ntp, and icmp protocols. (eth0 interface in the example).

In the iptables file, each append rule command (lines starting with -A) is a single line.

The example rules should be adjusted for your configuration. For example:

  • The append command, the -A lines and connection parameter -i should match the connectors for your hosts.
  • the CIDR network mask information for the source parameter -s should match the IP addresses for your network.

Example Master and Standby Master iptables Rules

Example iptables rules with comments for the /etc/sysconfig/iptables file on the Greenplum Database master host and standby master host.

# Following 3 are default rules. If the packet passes through
# the rule set it gets these rule.
# Drop all inbound packets by default.
# Drop all forwarded (routed) packets.
# Let anything outbound go through.
# Accept anything on the loopback interface.
-A INPUT -i lo -j ACCEPT
# If a connection has already been established allow the
# remote host packets for the connection to pass through.
# These rules let all tcp and udp through on the standard
# interconnect IP addresses and on the interconnect interfaces.
# NOTE: gpsyncmaster uses random tcp ports in the range 1025 to 65535
# and Greenplum Database uses random udp ports in the range 1025 to 65535.
-A INPUT -i eth4 -p udp -s -j ACCEPT
-A INPUT -i eth5 -p udp -s -j ACCEPT
-A INPUT -i eth4 -p tcp -s -j ACCEPT --syn -m state --state NEW
-A INPUT -i eth5 -p tcp -s -j ACCEPT --syn -m state --state NEW
# Allow snmp connections on the admin network on Greenplum DCA.
-A INPUT -i eth0 -p udp --dport snmp -s -j ACCEPT
-A INPUT -i eth0 -p tcp --dport snmp -s -j ACCEPT --syn -m state --state NEW
# Allow udp/tcp ntp connections on the admin network on Greenplum DCA.
-A INPUT -i eth0 -p udp --dport ntp -s -j ACCEPT
-A INPUT -i eth0 -p tcp --dport ntp -s -j ACCEPT --syn -m state --state NEW
# Allow ssh on all networks (This rule can be more strict).
-A INPUT -p tcp --dport ssh -j ACCEPT --syn -m state
--state NEW
# Allow Greenplum Database on all networks.
-A INPUT -p tcp --dport postgres -j ACCEPT --syn -m state --state NEW
# Allow Greenplum Command Center on the customer facing network.
-A INPUT -i eth1 -p tcp --dport 28080 -j ACCEPT --syn -m state --state NEW
# Allow ping and any other icmp traffic on the interconnect networks.
-A INPUT -i eth4 -p icmp -s -j ACCEPT
-A INPUT -i eth5 -p icmp -s -j ACCEPT
# Allow ping only on the admin network on Greenplum DCA.
-A INPUT -i eth0 -p icmp --icmp-type echo-request -s -j ACCEPT
# Log an error if a packet passes through the rules to the default
# INPUT rule (a DROP).
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

Example Segment Host iptables Rules

Example iptables rules for the /etc/sysconfig/iptables file on the Greenplum Database segment hosts. The rules for segment hosts are similar to the master rules with fewer interfaces and and fewer udp and tcp services.

-A INPUT -i lo -j ACCEPT
-A INPUT -i eth2 -p udp -s -j ACCEPT
-A INPUT -i eth3 -p udp -s -j ACCEPT
-A INPUT -i eth2 -p tcp -s -j ACCEPT --syn -m state --state NEW
-A INPUT -i eth3 -p tcp -s -j ACCEPT --syn -m state --state NEW
-A INPUT -i eth0 -p udp --dport snmp -s -j ACCEPT
-A INPUT -i eth0 -p tcp --dport snmp -j ACCEPT --syn -m state --state NEW
-A INPUT -p tcp --dport ssh -j ACCEPT --syn -m state --state NEW
-A INPUT -i eth2 -p icmp -s -j ACCEPT
-A INPUT -i eth3 -p icmp -s -j ACCEPT
-A INPUT -i eth0 -p icmp --icmp-type echo-request -s -j ACCEPT
-A INPUT -m limit --limit 5/min -j LOG --log-prefix "iptables denied: " --log-level 7

Next Steps

After you have configured the operating system environment and installed the Greenplum Database software on all of the hosts in the system, the next steps are: