Setting the Greenplum Recommended OS Parameters

Setting the Greenplum Recommended OS Parameters

Greenplum requires the certain operating system (OS) parameters be set on all hosts in your Greenplum Database system (masters and segments).

In general, the following categories of system parameters need to be altered:

  • Shared Memory - A Greenplum Database instance will not work unless the shared memory segment for your kernel is properly sized. Most default OS installations have the shared memory values set too low for Greenplum Database. On Linux systems, you must also disable the OOM (out of memory) killer. For information about Greenplum Database shared memory requirements, see the Greenplum Database server configuration parameter shared_buffers in the Greenplum Database Reference Guide.
  • Network - On high-volume Greenplum Database systems, certain network-related tuning parameters must be set to optimize network connections made by the Greenplum interconnect.
  • User Limits - User limits control the resources available to processes started by a user's shell. Greenplum Database requires a higher limit on the allowed number of file descriptors that a single process can have open. The default settings may cause some Greenplum Database queries to fail because they will run out of file descriptors needed to process the query.

Linux System Settings

  • Set the following parameters in the /etc/sysctl.conf file and reboot:
    kernel.shmmax = 500000000
    kernel.shmmni = 4096
    kernel.shmall = 4000000000
    kernel.sem = 250 512000 100 2048
    kernel.sysrq = 1
    kernel.core_uses_pid = 1
    kernel.msgmnb = 65536
    kernel.msgmax = 65536
    kernel.msgmni = 2048
    net.ipv4.tcp_syncookies = 1
    net.ipv4.ip_forward = 0
    net.ipv4.conf.default.accept_source_route = 0
    net.ipv4.tcp_tw_recycle = 1
    net.ipv4.tcp_max_syn_backlog = 4096
    net.ipv4.conf.all.arp_filter = 1
    net.ipv4.ip_local_port_range = 1025 65535
    net.core.netdev_max_backlog = 10000
    net.core.rmem_max = 2097152
    net.core.wmem_max = 2097152
    vm.overcommit_memory = 2
  • Set the following parameters in the /etc/security/limits.conf file:
    * soft nofile 65536
    * hard nofile 65536
    * soft nproc 131072
    * hard nproc 131072

    For Red Hat Enterprise Linux (RHEL) 6.x and CentOS 6.x, parameter values in the /etc/security/limits.d/90-nproc.conf file override the values in the limits.conf file. If a parameter value is set in both conf files, ensure that the parameter is set properly in the 90-nproc.conf file. The Linux module pam_limits sets user limits by reading the values from the limits.conf file and then from the 90-nproc.conf file. For information about PAM and user limits, see the documentation on PAM and pam_limits.

  • XFS is the preferred file system on Linux platforms for data storage. Greenplum recommends the following XFS mount options:
    rw,noatime,inode64,allocsize=16m

    See the manual page (man) for the mount command for more information about using that command (man mount opens the man page).

    The XFS options can also be set in the /etc/fstab file. This example entry from an fstab file specifies the XFS options.

    /dev/data /data xfs nodev,noatime,inode64,allocsize=16m 0 0
  • The Linux disk I/O scheduler for disk access supports different policies, such as CFQ, AS, and deadline.

    Pivotal recommends the following scheduler option: deadline. To specify a scheduler until the next system reboot, run the following:

    # echo schedulername > /sys/block/devname/queue/scheduler

    For example:

    # echo deadline > /sys/block/sbd/queue/scheduler

    You can specify the I/O scheduler at boot time with the elevator kernel parameter. Add the parameter elevator=deadline to the kernel command in the file /boot/grub/grub.conf, the GRUB boot loader configuration file. This is an example kernel command from a grub.conf file on RHEL 6.x or CentOS 6.x. The command is on multiple lines for readability.

    kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/
        elevator=deadline crashkernel=128M@16M  quiet console=tty1
        console=ttyS1,115200 panic=30 transparent_hugepage=never 
        initrd /initrd-2.6.18-274.3.1.el5.img
    To specify the I/O scheduler at boot time on On systems that use grub2 such as RHEL 7.x or CentOS 7.x, use the system utility grubby. This command adds the parameter when run as root.
    # grubby --update-kernel=ALL --args="elevator=deadline"

    After adding the parameter, reboot the system.

    For more information about the grubby utility, see your operating system documentation.

  • Each disk device file should have a read-ahead (blockdev) value of 16384.

    To verify the read-ahead value of a disk device:

    # /sbin/blockdev --getra devname

    For example:

    # /sbin/blockdev --getra /dev/sdb

    To set blockdev (read-ahead) on a device:

    # /sbin/blockdev --setra bytes devname

    For example:

    # /sbin/blockdev --setra 16384 /dev/sdb

    See the manual page (man) for the blockdev command for more information about using that command (man blockdev opens the man page).

  • Edit the /etc/hosts file and make sure that it includes the host names and all interface address names for every machine participating in your Greenplum Database system.
  • Disable Transparent Huge Pages (THP). RHEL 6.0 or higher enables THP by default. THP degrades Greenplum Database performance. One way to disable THP on RHEL 6.x is by adding the parameter transparent_hugepage=never to the kernel command in the file /boot/grub/grub.conf, the GRUB boot loader configuration file. This is an example kernel command from a grub.conf file. The command is on multiple lines for readability:
    kernel /vmlinuz-2.6.18-274.3.1.el5 ro root=LABEL=/
        elevator=deadline crashkernel=128M@16M  quiet console=tty1
        console=ttyS1,115200 panic=30 transparent_hugepage=never 
        initrd /initrd-2.6.18-274.3.1.el5.img
    On systems that use grub2 such as RHEL 7.x or CentOS 7.x, use the system utility grubby. This command adds the parameter when run as root.
    # grubby --update-kernel=ALL --args="transparent_hugepage=never"

    After adding the parameter, reboot the system.

    This cat command checks the state of THP. The output indicates that THP is disabled.
    $ cat /sys/kernel/mm/*transparent_hugepage/enabled
    always [never]

    For more information about Transparent Huge Pages or the grubby utility, see your operating system documentation.

  • Disable IPC object removal for RHEL 7.2 or CentOS 7.2. The default systemd setting RemoveIPC=yes removes IPC connections when non-system users log out. This causes the Greenplum Database utility gpinitsystem to fail with semaphore errors. Perform one of the following to avoid this issue.
    • Create the gpadmin user as a system account. For the useradd command, the -r option creates a user as a system user and the -m option creates a home directory for the user.
    • Disable RemoveIPC. Set this parameter in /etc/systemd/logind.conf on the Greenplum Database host systems.
      RemoveIPC=no

      The setting takes effect after restarting the systemd-login service or rebooting the system. To restart the service, run this command as the root user.

      service systemd-logind restart

Mac OS X System Settings

  • Add the following to /etc/sysctl.conf:
    kern.sysv.shmmax=2147483648 
    kern.sysv.shmmin=1 
    kern.sysv.shmmni=64 
    kern.sysv.shmseg=16 
    kern.sysv.shmall=524288 
    kern.maxfiles=65535 
    kern.maxfilesperproc=65535 
    net.inet.tcp.msl=60
  • Add the following line to /etc/hostconfig:
    HOSTNAME="your_hostname"