In PL/Container version 1.2.0 and later, you can use Greenplum Database resource groups to manage and limit the total CPU and memory resources of containers in PL/Container runtimes. For more information about enabling, configuring, and using Greenplum Database resource groups, refer to Using Resource Groups in the Greenplum Database Administrator Guide.

Resource groups for external components such as PL/Container use Linux control groups (cgroups) to manage component-level use of memory and CPU resources. When you manage PL/Container resources with resource groups, you configure both a memory limit and a CPU limit that Greenplum Database applies to all container instances that share the same PL/Container runtime configuration.

CREATE RESOURCE GROUP plpy_run1_rg WITH (MEMORY_AUDITOR=cgroup, CONCURRENCY=0,
     CPU_RATE_LIMIT=10, MEMORY_LIMIT=10);

PL/Container does not use the MEMORY_SHARED_QUOTA and MEMORY_SPILL_RATIO resource group memory limits. Refer to the CREATE RESOURCE GROUP reference page for detailed information about this SQL command.

SELECT groupname, groupid FROM gp_toolkit.gp_resgroup_config
     WHERE groupname='plpy_run1_rg';

  groupname   |  groupid  
--------------+----------
 plpy_run1_rg |   16391
(1 row)

plcontainer runtime-add -r python_run1 -i pivotaldata/plcontainer_python_shared:devel -l python -s resource_group_id=16391

You can also assign a resource group to a PL/Container runtime using the plcontainer runtime-edit command. For information about the plcontainer command, see The plcontainer Utility.

After you assign a resource group to a PL/Container runtime, all container instances that share the same runtime configuration are subject to the memory limit and the CPU limit that you configured for the group. If you decrease the memory limit of a PL/Container resource group, queries executing in running containers in the group may fail with an out of memory error. If you drop a PL/Container resource group while there are running container instances, Greenplum Database kills the running containers.

To use Greenplum Database resource groups to manage PL/Container resources, you must explicitly configure both resource groups and PL/Container.

Install the PL/Container language extension with the Greenplum Database gppkg utility.

1. Copy the PL/Container language extension package to the Greenplum Database master host as the gpadmin user.
2. Make sure Greenplum Database is up and running. If not, bring it up with this command.

   gpstart -a

3. Run the package installation command.

   gppkg -i plcontainer-1.1.0-rhel7-x86_64.gppkg

4. Source the file $GPHOME/greenplum_path.sh.

   source $GPHOME/greenplum_path.sh

5. Restart Greenplum Database.

   gpstop -ra

6. Enable PL/Container for specific databases by running this command.
   1. For PL/Container 1.1 and later, log into the database as a Greenplum Database superuser (gpadmin) and run this command.

      CREATE EXTENSION plcontainer; 

      The command registers PL/Container and creates PL/Container-specific functions and views.

   2. For PL/Container 1.0, run this command.

      psql -d your_database -f $GPHOME/share/postgresql/plcontainer/plcontainer_install.sql

      The SQL script registers the language plcontainer in the database and creates PL/Container-specific functions and views.

After installing PL/Container, you can manage Docker images and manage the PL/Container configuration with the Greenplum Database plcontainer utility.

To upgrade to version 1.1 or higher, uninstall version 1.0 and install the new version. The gppkg utility installs PL/Container version 1.1 and later as a Greenplum Database extension, while PL/Container 1.0 is installed as a Greenplum Database language. The Docker images and the PL/Container configuration do not change when upgrading PL/Container, only the PL/Container extension installation changes.

As part of the upgrade process, you must drop PL/Container from all databases that are configured with PL/Container.

SELECT proname, prosrc FROM pg_proc WHERE prolang = (SELECT oid FROM pg_language WHERE lanname = 'plcontainer');

After upgrading PL/Container for a database, re-install any user-created PL/Container UDFs that are required.

To upgrade from PL/Container version 1.1 or higher, you save the current configuration, upgrade PL/Container, and then restore the configuration after upgrade. There is no need to update the Docker images when you upgrade PL/Container.

On the Greenplum Database hosts, uninstall the Docker containers and images that are no longer required.

The plcontainer image-list command lists the Docker images that are installed on the local Greenplum Database host.

The plcontainer image-delete command deletes a specified Docker image from all Greenplum Database hosts.

For a database that no long requires PL/Container, remove support for PL/Container.

psql -d mytest -c 'DROP EXTENSION plcontainer cascade;'

psql -d mytest -f $GPHOME/share/postgresql/plcontainer/plcontainer_uninstall.sql

If no databases have plcontainer as a registered language, uninstall the Greenplum Database PL/Container language extension with the gppkg utility.

1. Use the Greenplum Database gppkg utility with the -r option to uninstall the PL/Container language extension. This example uninstalls the PL/Container language extension on a Linux system:

   $ gppkg -r plcontainer-1.1.0

   You can run the gppkg utility with the options -q --all to list the installed extensions and their versions.

2. Reload greenplum_path.sh.

   $ source $GPHOME/greenplum_path.sh

3. Restart the database.

   $ gpstop -ra

The values in the # container lines of the examples, plc_python_shared and plc_r_shared, are the id XML elements defined in the plcontainer_config.xml file. The id element is mapped to the image element that specifies the Docker image to be started. If you configured PL/Container with a different ID, change the value of the # container line. For information about configuring PL/Container and viewing the configuration settings, see Configuring PL/Container.

CREATE OR REPLACE FUNCTION pylog100() RETURNS double precision AS $$
# container: plc_python_shared
import math
return math.log10(100)
$$ LANGUAGE plcontainer;

CREATE OR REPLACE FUNCTION rlog100() RETURNS text AS $$
# container: plc_r_shared
return(log10(100))
$$ LANGUAGE plcontainer;

If the # container line in a UDF specifies an ID that is not in the PL/Container configuration file, Greenplum Database returns an error when you try to execute the UDF.

The plcontainer utility installs Docker images and manages the PL/Container configuration. The utility consists of two sets of commands.

• image-* commands manage Docker images on the Greenplum Database system hosts.
• runtime-* commands manage the PL/Container configuration file on the Greenplum Database instances. You can add Docker image information to the PL/Container configuration file including the image name, location, and shared folder information. You can also edit the configuration file.

To configure PL/Container to use a Docker image, you install the Docker image on all the Greenplum Database hosts and then add configuration information to the PL/Container configuration.

PL/Container configuration values, such as image names, runtime IDs, and parameter values and names are case sensitive.

plcontainer [command] [-h | --help]  [--verbose]

  image-add {{-f | --file} image_file} | {{-u | --URL} image_URL}
  image-delete {-i | --image} image_name
  image-list

  runtime-add {-r | --runtime} runtime_id
     {-i | --image} image_name {-l | --language} {python | r}
     [{-v | --volume} shared_volume [{-v| --volume} shared_volume...]]
     [{-s | --setting} param=value [{-s | --setting} param=value ...]]
  runtime-replace {-r | --runtime} runtime_id
     {-i | --image} image_name -l {r | python}
     [{-v | --volume} shared_volume [{-v | --volume} shared_volume...]]
     [{-s | --setting} param=value [{-s | --setting} param=value ...]]
  runtime-show {-r | --runtime} runtime_id
  runtime-delete {-r | --runtime} runtime_id
  runtime-edit [{-e | --editor} editor]
  runtime-backup {-f | --file} config_file
  runtime-restore {-f | --file} config_file
  runtime-verify

image-add location

Install a Docker image on the Greenplum Database hosts. Specify either the location of the Docker image file on the host or the URL to the Docker image. These are the supported location options.

• {-f | --file} image_file Specify the tar archive file on the host that contains the Docker image. This example points to an image file in the gpadmin home directory /home/gpadmin/test_image.tar.gz
• {-u | --URL} image_URL Specify the URL of the Docker repository and image. This example URL points to a local Docker repository 192.168.0.1:5000/images/mytest_plc_r:devel

After installing the Docker image, use the runtime-add command to configure PL/Container to use the Docker image.

image-delete {-i | --image} image_name

Remove an installed Docker image from all Greenplum Database hosts. Specify the full Docker image name including the tag for example pivotaldata/plcontainer_python_shared:1.0.0

image-list

List the Docker images installed on the host. The command list only the images on the local host, not remote hosts. The command lists all installed Docker images, including images installed with Docker commands.

runtime-add options

Add configuration information to the PL/Container configuration file on all Greenplum Database hosts. If the specified runtime_id exists, the utility returns an error and the configuration information is not added.

For information about PL/Container configuration, see PL/Container Configuration File.

These are the supported options:

{-i | --image} docker-image

Required. Specify the full Docker image name, including the tag, that is installed on the Greenplum Database hosts. For example pivotaldata/plcontainer_python:1.0.0.

The utility returns a warning if the specified Docker image is not installed.

The plcontainer image-list command displays installed image information including the name and tag (the Repository and Tag columns).

{-l | --language} python | r

Required. Specify the PL/Container language type, supported values are python (PL/Python) and r (PL/R). When adding configuration information for a new runtime, the utility adds a startup command to the configuration based on the language you specify.

Startup command for the Python language.

/clientdir/pyclient.sh

Startup command for the R language.

/clientdir/rclient.sh

{-r | --runtime} runtime_id

Required. Add the runtime ID. When adding a runtime element in the PL/Container configuration file, this is the value of the id element in the PL/Container configuration file. Maximum length is 63 Bytes.

You specify the name in the Greenplum Database UDF on the # container line. See Examples.

{-s | --setting} param=value

Optional. Specify a setting to add to the runtime configuration information. You can specify this option multiple times. The setting applies to the runtime configuration specified by the runtime_id. The parameter is the XML attribute of the settings element in the PL/Container configuration file. These are valid parameters.

• cpu_share - Set the CPU limit for each container in the runtime configuration. The default value is 1024. The value is a relative weighting of CPU usage compared to other containers.
• memory_mb - Set the memory limit for each container in the runtime configuration. The default value is 1024. The value is an integer that specifies the amount of memory in MB.
• resource_group_id - Assign the specified resource group to the runtime configuration. The resource group limits the total CPU and memory resource usage for all containers that share this runtime configuration. You must specify the groupid of the resource group. For information about managing PL/Container resources, see About PL/Container Resource Management.
• roles - Specify the Greenplum Database roles that are allowed to run a container for the runtime configuration. You can specify a single role name or comma separated lists of role names. The default is no restriction.
• use_container_logging - Enable or disable Docker logging for the container. The value is either yes (enable logging) or no (disable logging, the default).

  The Greenplum Database server configuration parameter log_min_messages controls the log level. The default log level is warning. For information about PL/Container log information, see Notes.

{-v | --volume} shared-volume

Optional. Specify a Docker volume to bind mount. You can specify this option multiple times to define multiple volumes.

The format for a shared volume: host-dir:container-dir:[rw|ro]. The information is stored as attributes in the shared_directory element of the runtime element in the PL/Container configuration file.

• host-dir - absolute path to a directory on the host system. The Greenplum Database administrator user (gpadmin) must have appropriate access to the directory.
• container-dir - absolute path to a directory in the Docker container.
• [rw|ro] - read-write or read-only access to the host directory from the container.

When adding configuration information for a new runtime, the utility adds this read-only shared volume information.

greenplum-home/bin/plcontainer_clients:/clientdir:ro

If needed, you can specify other shared directories. The utility returns an error if the specified container-dir is the same as the one that is added by the utility, or if you specify multiple shared volumes with the same container-dir.

Warning: Allowing read-write access to a host directory requires special considerations.

• When specifying read-write access to host directory, ensure that the specified host directory has the correct permissions.
• When running PL/Container user-defined functions, multiple concurrent Docker containers that are running on a host could change data in the host directory. Ensure that the functions support multiple concurrent access to the data in the host directory.

runtime-backup {-f | --file} config_file

Copies the PL/Container configuration file to the specified file on the local host.

runtime-delete {-r | --runtime} runtime_id

Removes runtime configuration information in the PL/Container configuration file on all Greenplum Database instances. The utility returns a message if the specified runtime_id does not exist in the file.

runtime-edit [{-e | --editor} editor]

Edit the XML file plcontainer_configuration.xml with the specified editor. The default editor is vi.

Saving the file updates the configuration file on all Greenplum Database hosts. If errors exist in the updated file, the utility returns an error and does not update the file.

runtime-replace options

Replaces runtime configuration information in the PL/Container configuration file on all Greenplum Database instances. If the runtime_id does not exist, the information is added to the configuration file. The utility adds a startup command and shared directory to the configuration.

See runtime-add for command options and information added to the configuration.

runtime-restore {-f | --file} config_file

Replaces information in the PL/Container configuration file plcontainer_configuration.xml on all Greenplum Database instances with the information from the specified file on the local host.

runtime-show [{-r | --runtime} runtime_id]

Displays formatted PL/Container runtime configuration information. If a runtime_id is not specified, the configuration for all runtime IDs are displayed.

runtime-verify

Checks the PL/Container configuration information on the Greenplum Database instances with the configuration information on the master. If the utility finds inconsistencies, you are prompted to replace the remote copy with the local copy. The utility also performs XML validation.

-h | --help

Display help text. If specified without a command, displays help for all plcontainer commands. If specified with a command, displays help for the command.

--verbose

Enable verbose logging for the command.

PL/Container maintains a configuration file plcontainer_configuration.xml in the data directory of all Greenplum Database segments. The PL/Container configuration file is an XML file. In the XML file, the root element configuration contains one or more runtime elements. You specify the id of the runtime element in the # container: line of a PL/Container function definition.

In an XML file, names, such as element and attribute names, and values are case sensitive.

<?xml version="1.0" ?>
<configuration>
    <runtime>
        <id>plc_python_example1</id>
        <image>pivotaldata/plcontainer_python_with_clients:0.1</image>
        <command>./pyclient</command>
    </runtime>
    <runtime>
        <id>plc_python_example2</id>
        <image>pivotaldata/plcontainer_python_without_clients:0.1</image>
        <command>/clientdir/pyclient.sh</command>
        <shared_directory access="ro" container="/clientdir" host="/usr/local/greenplum-db/bin/plcontainer_clients"/>
        <setting memory_mb="512"/>
        <setting use_container_logging="yes"/>
        <setting cpu_share="1024"/>
        <setting resource_group_id="16391"/>
    </runtime>
    <runtime>
        <id>plc_r_example</id>
        <image>pivotaldata/plcontainer_r_without_clients:0.2</image>
        <command>/clientdir/rclient.sh</command>
        <shared_directory access="ro" container="/clientdir" host="/usr/local/greenplum-db/bin/plcontainer_clients"/>
        <setting use_container_logging="yes"/>
        <setting roles="gpadmin,user1"/>
    </runtime>
    <runtime>
</configuration>

These are the XML elements and attributes in a PL/Container configuration file.

configuration

Root element for the XML file.

runtime

One element for each specific container available in the system. These are child elements of the configuration element.

id

Required. The value is used to reference a Docker container from a PL/Container user-defined function. The id value must be unique in the configuration. The id must start with a character or digit (a-z, A-Z, or 0-9) and can contain characters, digits, or the characters _ (underscore), . (period), or - (dash). Maximum length is 63 Bytes.

The id specifies which Docker image to use when PL/Container creates a Docker container to execute a user-defined function.

image

Required. The value is the full Docker image name, including image tag. The same way you specify them for starting this container in Docker. Configuration allows to have many container objects referencing the same image name, this way in Docker they would be represented by identical containers.

For example, you might have two runtime elements, with different id elements, plc_python_128 and plc_python_256, both referencing the Docker image pivotaldata/plcontainer_python:1.0.0. The first runtime specifies a 128MB RAM limit and the second one specifies a 256MB limit that is specified by the memory_mb attribute of a setting element.

command

Required. The value is the command to be run inside of container to start the client process inside in the container. When creating a runtime element, the plcontainer utility adds a command element based on the language (the -l option).

command element for the python language.

<command>/clientdir/pyclient.sh</command>

command element for the R language.

<command>/clientdir/rclient.sh</command>

You should modify the value only if you build a custom container and want to implement some additional initialization logic before the container starts.

Note: This element cannot be set with the plcontainer utility. You can update the configuration file with the plcontainer runtime-edit command.

shared_directory

Optional. This element specifies a shared Docker shared volume for a container with access information. Multiple shared_directory elements are allowed. Each shared_directory element specifies a single shared volume. XML attributes for the shared_directory element:

• host - a directory location on the host system.
• container - a directory location inside of container.
• access - access level to the host directory, which can be either ro (read-only) or rw (read-write).

When creating a runtime element, the plcontainer utility adds a shared_directory element.

<shared_directory access="ro" container="/clientdir" host="/usr/local/greenplum-db/bin/plcontainer_clients"/>

For each runtime element, the container attribute of the shared_directory elements must be unique. For example, a runtime element cannot have two shared_directory elements with attribute container="/clientdir".

Warning: Allowing read-write access to a host directory requires special consideration.

• When specifying read-write access to host directory, ensure that the specified host directory has the correct permissions.
• When running PL/Container user-defined functions, multiple concurrent Docker containers that are running on a host could change data in the host directory. Ensure that the functions support multiple concurrent access to the data in the host directory.

settings

Optional. This element specifies Docker container configuration information. Each setting element contains one attribute. The element attribute specifies logging, memory, or networking information. For example, this element enables logging.

<setting use_container_logging="yes"/>

These are the valid attributes.

cpu_share

Optional. Specify the CPU usage for each PL/Container container in the runtime. The value of the element is a positive integer. The default value is 1024. The value is a relative weighting of CPU usage compared to other containers.

For example, a container with a cpu_share of 2048 is allocated double the CPU slice time compared with container with the default value of 1024.

memory_mb="size"

Optional. The value specifies the amount of memory, in MB, that each container is allowed to use. Each container starts with this amount of RAM and twice the amount of swap space. The container memory consumption is limited by the host system cgroups configuration, which means in case of memory overcommit, the container is terminated by the system.

resource_group_id="rg_groupid"

Optional. The value specifies the groupid of the resource group to assign to the PL/Container runtime. The resource group limits the total CPU and memory resource usage for all running containers that share this runtime configuration. You must specify the groupid of the resource group. If you do not assign a resource group to a PL/Container runtime configuration, its container instances are limited only by system resources. For information about managing PL/Container resources, see About PL/Container Resource Management.

roles="list_of_roles"

Optional. The value is a Greenplum Database role name or a comma-separated list of roles. PL/Container runs a container that uses the PL/Container runtime configuration only for the listed roles. If the attribute is not specified, any Greenplum Database role can run an instance of this container runtime configuration. For example, you create a UDF that specifies the plcontainer language and identifies a # container: runtime configuration that has the roles attribute set. When a role (user) runs the UDF, PL/Container checks the list of roles and runs the container only if the role is on the list.

use_container_logging="{yes | no}"

Optional. Enables or disables Docker logging for the container. The attribute value yes enables logging. The attribute value no disables logging (the default).

The Greenplum Database server configuration parameter log_min_messages controls the PL/Container log level. The default log level is warning. For information about PL/Container log information, see Notes.

By default, the PL/Container log information is sent to a system service. On Red Hat 7 or CentOS 7 systems, the log information is sent to the journald service. On Red Hat 6 or CentOS 6 systems, the log is sent to the syslogd service.

You can add a runtime element to the PL/Container configuration file with the plcontainer runtime-add command. The command options specify information such as the runtime ID, Docker image, and language. You can use the plcontainer runtime-replace command to update an existing runtime element. The utility updates the configuration file on the master and all segment instances.

The PL/Container configuration file can contain multiple runtime elements that reference the same Docker image specified by the XML element image. In the example configuration file, the runtime elements contain id elements named plc_python_128 and plc_python_256, both referencing the Docker container pivotaldata/plcontainer_python:1.0.0. The first runtime element is defined with a 128MB RAM limit and the second one with a 256MB RAM limit.

<configuration>
  <runtime>
    <id>plc_python_128</id>
    <image>pivotaldata/plcontainer_python:1.0.0</image>
    <command>./client</command>
    <shared_directory access="ro" container="/clientdir" host="/usr/local/gpdb/bin/plcontainer_clients"/>
    <setting memory_mb="128"/>
  </runtime>
  <runtime>
    <id>plc_python_256</id>
    <image>pivotaldata/plcontainer_python:1.0.0</image>
    <command>./client</command>
    <shared_directory access="ro" container="/clientdir" host="/usr/local/gpdb/bin/plcontainer_clients"/>
    <setting memory_mb="256"/>
    <setting resource_group_id="16391"/>
  </runtime>
<configuration>

• PL/Container does not support the Greenplum Database domain object.
• PL/Container maintains the configuration file plcontainer_configuration.xml in the data directory of all Greenplum Database segment instances: master, standby master, primary, and mirror. This query lists the Greenplum Database system data directories:

  SELECT hostname, datadir FROM gp_segment_configuration;

  A sample PL/Container configuration file is in $GPHOME/share/postgresql/plcontainer.

• When Greenplum Database executes a PL/Container UDF, Query Executer (QE) processes start Docker containers and reuse them as needed. After a certain amount of idle time, a QE process quits and destroys its Docker containers. You can control the amount of idle time with the Greenplum Database server configuration parameter gp_vmem_idle_resource_timeout. Controlling the idle time might help with Docker container reuse and avoid the overhead of creating and starting a Docker container.
  Warning: Changing gp_vmem_idle_resource_timeout value, might affect performance due to resource issues. The parameter also controls the freeing of Greenplum Database resources other than Docker containers.
• If a PL/Container Docker container exceeds the maximum allowed memory, it is terminated and an out of memory warning is displayed. On Red Hat 6 or CentOS 6 systems that are configured with Docker version 1.7.1, the out of memory warning is also displayed if the PL/Container Docker container main program (PID 1) is terminated.
• In some cases, when PL/Container is running in a high concurrency environment, the Docker daemon hangs with log entries that indicate a memory shortage. This can happen even when the system seems to have adequate free memory.

  The issue seems to be triggered by a combination of two factors, the aggressive virtual memory requirement of the Go language (golang) runtime that is used by PL/Container, and the Greenplum Database Linux server kernel parameter setting for overcommit_memory. The parameter is set to 2 which does not allow memory overcommit.

  A workaround that might help is to increase the amount of swap space and increase the Linux server kernel parameter overcommit_ratio. If the issue still occurs after the changes, there might be memory shortage. You should check free memory on the system and add more RAM if needed. You can also decrease the cluster load.

• PL/Container does not limit the Docker base device size, the size of the Docker container. In some cases, the Docker daemon controls the base device size. For example, if the Docker storage driver is devicemapper, the Docker daemon --storage-opt option flag dm.basesize controls the base device size. The default base device size for devicemapper is 10GB. The Docker command docker info displays Docker system information including the storage driver. The base device size is displayed in Docker 1.12 and later. For information about Docker storage drivers, see the Docker information Daemon storage-driver.

  When setting the Docker base device size, the size must be set on all Greenplum Database hosts.

• When PL/Container logging is enabled, you can set the log level with the Greenplum Database server configuration parameter log_min_messages. The default log level is warning. The parameter controls the PL/Container log level and also controls the Greenplum Database log level.
  • PL/Container logging is enabled or disabled for each runtime ID with the setting attribute use_container_logging. The default is no logging.
  • The PL/Container log information is the information from the UDF that is run in the Docker container. By default, the PL/Container log information is sent to a system service. On Red Hat 7 or CentOS 7 systems, the log information is sent to the journald service. On Red Hat 6 or CentOS 6 systems, the log information is sent to the syslogd service. The PL/Container log information is sent to the log file of the host were the Docker container runs.
  • The Greenplum Database log information is sent to log file on the Greenplum Database master.

  When testing or troubleshooting a PL/Container UDF, you can change the Greenplum Database log level with the SET command. You can set the parameter in the session before you run your PL/Container UDF. This example sets the log level to debug1.

  SET log_min_messages='debug1' ;

  Note: The parameter log_min_messages controls both the Greenplum Database and PL/Container logging, increasing the log level might affect Greenplum Database performance even if a PL/Container UDF is not running.