Installing the Data Science Packages
Installing the Data Science Packages
Information about installing the Greenplum Database Python and R Data Science Packages.
Doc Index
- Pivotal Greenplum® Database 4.3.33 Documentation
- Release Notes
-
Installation Guide
- Preface
- Introduction to Greenplum
- Estimating Storage Capacity
-
Configuring Your Systems and Installing Greenplum
- System Requirements
- Setting the Greenplum Recommended OS Parameters
- Running the Greenplum Installer
- Installing and Configuring Greenplum on all Hosts
- Installing Oracle Compatibility Functions
- Installing Greenplum Database Extensions
- Creating the Data Storage Areas
- Synchronizing System Clocks
- Enabling iptables
- Amazon EC2 Configuration (Amazon Web Services)
- Next Steps
- Installing the Data Science Packages
- Validating Your Systems
- Configuring Localization Settings
- Initializing a Greenplum Database System
- Installation Management Utilities
- Greenplum Environment Variables
-
Administrator Guide
- Greenplum Database Concepts
-
Managing a Greenplum System
- Starting and Stopping Greenplum Database
- Accessing the Database
-
Configuring the Greenplum Database System
- About Greenplum Database Master and Local Parameters
- Setting Configuration Parameters
- Viewing Server Configuration Parameter Settings
-
Configuration Parameter Categories
- Connection and Authentication Parameters
- System Resource Consumption Parameters
-
Query Tuning Parameters
- GPORCA Configuration Parameters
- Query Plan Operator Control Parameters
- Legacy Query Optimizer Costing Parameters
- Database Statistics Sampling Parameters
- Sort Operator Configuration Parameters
- Aggregate Operator Configuration Parameters
- Join Operator Configuration Parameters
- Other Legacy Query Optimizer Configuration Parameters
- Other Legacy Query Optimizer Configuration Parameter
- Error Reporting and Logging Parameters
- System Monitoring Parameters
- Runtime Statistics Collection Parameters
- Automatic Statistics Collection Parameters
- Client Connection Default Parameters
- Lock Management Parameters
- Workload Management Parameters
- External Table Parameters
- Database Table Parameters
- Database and Tablespace/Filespace Parameters
- Past PostgreSQL Version Compatibility Parameters
- Greenplum Array Configuration Parameters
- Greenplum Master and Segment Mirroring Parameters
- Greenplum Database Extension Parameters
- Enabling High Availability Features
-
Backing Up and Restoring Databases
- Backup and Restore Overview
-
Parallel Backup with gpcrondump and gpdbrestore
- Backup and Restore Options
- Backing Up with gpcrondump
- Backing Up a Set of Tables
- Creating Incremental Backups
- Backup Process and Locks
- Using Direct I/O
- Using Named Pipes
- Backing Up Databases with Data Domain Boost
- Backing Up Databases with Veritas NetBackup
- Restoring Greenplum Databases
- Restoring a Database Using gpdbrestore
- Restoring to a Different Greenplum System Configuration
-
Parallel Backup with gpbackup and gprestore
- Requirements and Limitations
- Objects Included in a Backup or Restore
- Performing Basic Backup and Restore Operations
- Filtering the Contents of a Backup or Restore
- Configuring Email Notifications
- Understanding Backup Files
- Creating Incremental Backups with gpbackup and gprestore
- Using gpbackup and gprestore with BoostFS
- Using gpbackup Storage Plugins
- Backup/Restore Storage Plugin API (Beta)
- Expanding a Greenplum System
- Migrating Data with gptransfer
- Monitoring a Greenplum System
- Routine System Maintenance Tasks
- Recommended Monitoring and Maintenance Tasks
- Managing Greenplum Database Access
-
Working with Databases
- Defining Database Objects
- Distribution and Skew
- Managing Data
-
Loading and Unloading Data
- Working with File-Based External Tables
- Using the Greenplum Parallel File Server (gpfdist)
-
Using Hadoop Distributed File System (HDFS) Tables
-
One-time HDFS Protocol Installation
- Grant Privileges for the HDFS Protocol
- Specify HDFS Data in an External Table Definition
- HDFS Readable and Writable External Table Examples
- Reading and Writing Custom-Formatted HDFS Data
- Example 1 - Read Custom-Formatted Data from HDFS
- Sample MapReduce Code
- Example 2 - Write Custom-Formatted Data from Greenplum Database to HDFS
- MapReduce Sample Code
- About gphdfs JVM Memory
- Using Amazon EMR with Greenplum Database installed on AWS
- Support for Avro Files
- Support for Parquet Files
-
One-time HDFS Protocol Installation
- Creating and Using Web External Tables
- Loading Data Using an External Table
- Loading and Writing Non-HDFS Custom Data
-
Creating External Tables - Examples
- Example 1—Single gpfdist instance on single-NIC machine
- Example 2—Multiple gpfdist instances
- Example 3—Multiple gpfdists instances
- Example 4—Single gpfdist instance with error logging
- Example 5—TEXT Format on a Hadoop Distributed File Server
- Example 6—Multiple files in CSV format with header rows
- Example 7—Readable Web External Table with Script
- Example 8—Writable External Table with gpfdist
- Example 9—Writable External Web Table with Script
- Example 10—Readable and Writable External Tables with XML Transformations
- Handling Load Errors
- Loading Data with gpload
- Loading Data with COPY
- Running COPY in Single Row Error Isolation Mode
- Optimizing Data Load and Query Performance
- Unloading Data from Greenplum Database
- Transforming XML Data
- Formatting Data Files
- Example Custom Data Access Protocol
- Querying Data
- Managing Performance
-
Utility Guide
-
Management Utility Reference
- analyzedb
- gpactivatestandby
- gpaddmirrors
- gpbackup
- gpbitmapreindex
- gpcheck
- gpcheckcat
- gpcheckperf
- gpconfig
- gpcrondump
- gpdbrestore
- gpdeletesystem
- gpexpand
- gpfdist
- gpfilespace
- gpinitstandby
- gpinitsystem
- gpload
- gplogfilter
- gpmapreduce
- gpmfr
- gpmigrator
- gpmigrator_mirror
- gpmovemirrors
- gpperfmon_install
- gppkg
- gprecoverseg
- gpreload
- gprestore
- gpscp
- gpseginstall
- gpssh
- gpssh-exkeys
- gpstart
- gpstate
- gpstop
- gpsys1
- gptransfer
- pgbouncer
- pgbouncer.ini
- pgbouncer-admin
- Client Utility Reference
- Support and Diagnostic Utility Reference
- Oracle Compatibility Functions
-
Management Utility Reference
-
Reference Guide
- Preface
-
SQL Command Reference
- SQL Syntax Summary
- ABORT
- ALTER AGGREGATE
- ALTER CONVERSION
- ALTER DATABASE
- ALTER DOMAIN
- ALTER EXTERNAL TABLE
- ALTER FILESPACE
- ALTER FUNCTION
- ALTER GROUP
- ALTER INDEX
- ALTER LANGUAGE
- ALTER OPERATOR
- ALTER OPERATOR CLASS
- ALTER PROTOCOL
- ALTER RESOURCE QUEUE
- ALTER ROLE
- ALTER SCHEMA
- ALTER SEQUENCE
- ALTER TABLE
- ALTER TABLESPACE
- ALTER TYPE
- ALTER USER
- ANALYZE
- BEGIN
- CHECKPOINT
- CLOSE
- CLUSTER
- COMMENT
- COMMIT
- COPY
- CREATE AGGREGATE
- CREATE CAST
- CREATE CONVERSION
- CREATE DATABASE
- CREATE DOMAIN
- CREATE EXTERNAL TABLE
- CREATE FUNCTION
- CREATE GROUP
- CREATE INDEX
- CREATE LANGUAGE
- CREATE OPERATOR
- CREATE OPERATOR CLASS
- CREATE PROTOCOL
- CREATE RESOURCE QUEUE
- CREATE ROLE
- CREATE RULE
- CREATE SCHEMA
- CREATE SEQUENCE
- CREATE TABLE
- CREATE TABLE AS
- CREATE TABLESPACE
- CREATE TYPE
- CREATE USER
- CREATE VIEW
- DEALLOCATE
- DECLARE
- DELETE
- DROP AGGREGATE
- DROP CAST
- DROP CONVERSION
- DROP DATABASE
- DROP DOMAIN
- DROP EXTERNAL TABLE
- DROP FILESPACE
- DROP FUNCTION
- DROP GROUP
- DROP INDEX
- DROP LANGUAGE
- DROP OPERATOR
- DROP OPERATOR CLASS
- DROP OWNED
- DROP PROTOCOL
- DROP RESOURCE QUEUE
- DROP ROLE
- DROP RULE
- DROP SCHEMA
- DROP SEQUENCE
- DROP TABLE
- DROP TABLESPACE
- DROP TYPE
- DROP USER
- DROP VIEW
- END
- EXECUTE
- EXPLAIN
- FETCH
- GRANT
- INSERT
- LOAD
- LOCK
- MOVE
- PREPARE
- REASSIGN OWNED
- REINDEX
- RELEASE SAVEPOINT
- RESET
- REVOKE
- ROLLBACK
- ROLLBACK TO SAVEPOINT
- SAVEPOINT
- SELECT
- SELECT INTO
- SET
- SET ROLE
- SET SESSION AUTHORIZATION
- SET TRANSACTION
- SHOW
- START TRANSACTION
- TRUNCATE
- UPDATE
- VACUUM
- VALUES
- SQL 2008 Optional Feature Compliance
- Greenplum Environment Variables
-
System Catalog Reference
- System Tables
- System Views
-
System Catalogs Definitions
- gp_configuration_history
- gp_db_interfaces
- gp_distributed_log
- gp_distributed_xacts
- gp_distribution_policy
- gpexpand.expansion_progress
- gpexpand.status
- gpexpand.status_detail
- gp_fastsequence
- gp_fault_strategy
- gp_global_sequence
- gp_id
- gp_interfaces
- gp_persistent_database_node
- gp_persistent_filespace_node
- gp_persistent_relation_node
- gp_persistent_tablespace_node
- gp_pgdatabase
- gp_relation_node
- gp_resqueue_status
- gp_san_configuration
- gp_segment_configuration
- gp_transaction_log
- gp_version_at_initdb
- pg_aggregate
- pg_am
- pg_amop
- pg_amproc
- pg_appendonly
- pg_attrdef
- pg_attribute
- pg_attribute_encoding
- pg_auth_members
- pg_authid
- pg_cast
- pg_class
- pg_compression
- pg_constraint
- pg_conversion
- pg_database
- pg_depend
- pg_description
- pg_exttable
- pg_filespace
- pg_filespace_entry
- pg_index
- pg_inherits
- pg_language
- pg_largeobject
- pg_listener
- pg_locks
- pg_namespace
- pg_opclass
- pg_operator
- pg_partition
- pg_partition_columns
- pg_partition_encoding
- pg_partition_rule
- pg_partition_templates
- pg_partitions
- pg_pltemplate
- pg_proc
- pg_resourcetype
- pg_resqueue
- pg_resqueue_attributes
- pg_resqueuecapability
- pg_rewrite
- pg_roles
- pg_shdepend
- pg_shdescription
- pg_stat_activity
- pg_stat_last_operation
- pg_stat_last_shoperation
- pg_stat_operations
- pg_stat_partition_operations
- pg_stat_replication
- pg_statistic
- pg_stat_resqueues
- pg_tablespace
- pg_trigger
- pg_type
- pg_type_encoding
- pg_user_mapping
- pg_window
-
The gp_toolkit Administrative Schema
- Checking for Tables that Need Routine Maintenance
- Checking for Locks
- Checking Append-Optimized Tables
- Viewing Greenplum Database Server Log Files
- Checking Server Configuration Files
- Checking for Failed Segments
- Checking Resource Queue Activity and Status
- Checking Query Disk Spill Space Usage
- Viewing Users and Groups (Roles)
- Checking Database Object Sizes and Disk Space
- Checking for Uneven Data Distribution
- gpperfmon Database Reference
- Greenplum Database Data Types
- Character Set Support
-
Server Configuration Parameters
- Parameter Types and Values
- Setting Parameters
- Parameter Categories
-
Configuration Parameters
- add_missing_from
- application_name
- array_nulls
- authentication_timeout
- backslash_quote
- block_size
- bonjour_name
- check_function_bodies
- client_encoding
- client_min_messages
- cpu_index_tuple_cost
- cpu_operator_cost
- cpu_tuple_cost
- cursor_tuple_fraction
- custom_variable_classes
- DateStyle
- db_user_namespace
- deadlock_timeout
- debug_assertions
- debug_pretty_print
- debug_print_parse
- debug_print_plan
- debug_print_prelim_plan
- debug_print_rewritten
- debug_print_slice_table
- default_statistics_target
- default_tablespace
- default_transaction_isolation
- default_transaction_read_only
- dynamic_library_path
- effective_cache_size
- enable_bitmapscan
- enable_groupagg
- enable_hashagg
- enable_hashjoin
- enable_indexscan
- enable_mergejoin
- enable_nestloop
- enable_seqscan
- enable_sort
- enable_tidscan
- escape_string_warning
- explain_pretty_print
- extra_float_digits
- filerep_mirrorvalidation_during_resync
- from_collapse_limit
- gp_adjust_selectivity_for_outerjoins
- gp_analyze_relative_error
- gp_appendonly_compaction
- gp_appendonly_compaction_threshold
- gp_autostats_mode
- gp_autostats_mode_in_functions
- gp_autostats_on_change_threshold
- gp_backup_directIO
- gp_backup_directIO_read_chunk_mb
- gp_cached_segworkers_threshold
- gp_command_count
- gp_connectemc_mode
- gp_connection_send_timeout
- gp_connections_per_thread
- gp_content
- gp_create_table_random_default_distribution
- gp_dbid
- gp_debug_linger
- gp_default_storage_options
- gp_dynamic_partition_pruning
- gp_email_from
- gp_email_smtp_password
- gp_email_smtp_server
- gp_email_smtp_userid
- gp_email_to
- gp_enable_adaptive_nestloop
- gp_enable_agg_distinct
- gp_enable_agg_distinct_pruning
- gp_enable_direct_dispatch
- gp_enable_exchange_default_partition
- gp_enable_fallback_plan
- gp_enable_fast_sri
- gp_enable_gpperfmon
- gp_enable_groupext_distinct_gather
- gp_enable_groupext_distinct_pruning
- gp_enable_multiphase_agg
- gp_enable_predicate_propagation
- gp_enable_preunique
- gp_enable_relsize_collection
- gp_enable_segment_copy_checking
- gp_enable_sequential_window_plans (Beta)
- gp_enable_sort_distinct
- gp_enable_sort_limit
- gp_external_enable_exec
- gp_external_grant_privileges
- gp_external_max_segs
- gp_filerep_tcp_keepalives_count
- gp_filerep_tcp_keepalives_idle
- gp_filerep_tcp_keepalives_interval
- gp_fts_probe_interval
- gp_fts_probe_retries
- gp_fts_probe_threadcount
- gp_fts_probe_timeout
- gp_gpperfmon_send_interval
- gpperfmon_log_alert_level
- gp_hadoop_home
- gp_hadoop_target_version
- gp_hashjoin_tuples_per_bucket
- gp_idf_deduplicate
- gp_initial_bad_row_limit
- gp_inline_simple_cte
- gp_interconnect_fc_method
- gp_interconnect_hash_multiplier
- gp_interconnect_queue_depth
- gp_interconnect_setup_timeout
- gp_interconnect_snd_queue_depth
- gp_interconnect_type
- gp_log_format
- gp_log_fts
- gp_log_gang
- gp_max_csv_line_length
- gp_max_databases
- gp_max_filespaces
- gp_max_local_distributed_cache
- gp_max_packet_size
- gp_max_plan_size
- gp_max_tablespaces
- gp_motion_cost_per_row
- gp_num_contents_in_cluster
- gp_reject_percent_threshold
- gp_reraise_signal
- gp_resqueue_memory_policy
- gp_resqueue_priority
- gp_resqueue_priority_cpucores_per_segment
- gp_resqueue_priority_sweeper_interval
- gp_role
- gp_safefswritesize
- gp_segment_connect_timeout
- gp_segments_for_planner
- gp_session_id
- gp_set_proc_affinity
- gp_set_read_only
- gp_snmp_community
- gp_snmp_monitor_address
- gp_snmp_use_inform_or_trap
- gp_statistics_pullup_from_child_partition
- gp_statistics_use_fkeys
- gp_strict_xml_parse
- gp_vmem_idle_resource_timeout
- gp_vmem_protect_limit
- gp_vmem_protect_segworker_cache_limit
- gp_workfile_checksumming
- gp_workfile_compress_algorithm
- gp_workfile_limit_files_per_query
- gp_workfile_limit_per_query
- gp_workfile_limit_per_segment
- gpperfmon_port
- integer_datetimes
- IntervalStyle
- join_collapse_limit
- keep_wal_segments
- krb_caseins_users
- krb_server_keyfile
- krb_srvname
- lc_collate
- lc_ctype
- lc_messages
- lc_monetary
- lc_numeric
- lc_time
- listen_addresses
- local_preload_libraries
- log_autostats
- log_connections
- log_disconnections
- log_dispatch_stats
- log_duration
- log_error_verbosity
- log_executor_stats
- log_hostname
- log_min_duration_statement
- log_min_error_statement
- log_min_messages
- log_parser_stats
- log_planner_stats
- log_rotation_age
- log_rotation_size
- log_statement
- log_statement_stats
- log_timezone
- log_truncate_on_rotation
- max_appendonly_tables
- max_connections
- max_files_per_process
- max_fsm_pages
- max_fsm_relations
- max_function_args
- max_identifier_length
- max_index_keys
- max_locks_per_transaction
- max_prepared_transactions
- max_resource_portals_per_transaction
- max_resource_queues
- max_stack_depth
- max_statement_mem
- optimizer
- optimizer_analyze_root_partition
- optimizer_array_expansion_threshold
- optimizer_control
- optimizer_cte_inlining_bound
- optimizer_enable_associativity
- optimizer_enable_master_only_queries
- optimizer_force_multistage_agg
- optimizer_force_three_stage_scalar_dqa
- optimizer_join_arity_for_associativity_commutativity
- optimizer_join_order
- optimizer_join_order_threshold
- optimizer_minidump
- optimizer_nestloop_factor
- optimizer_parallel_union
- optimizer_print_missing_stats
- optimizer_print_optimization_stats
- optimizer_sort_factor
- password_encryption
- password_hash_algorithm
- pgcrypto.fips
- pgstat_track_activity_query_size
- pljava_classpath
- pljava_classpath_insecure
- pljava_statement_cache_size
- pljava_release_lingering_savepoints
- pljava_vmoptions
- port
- random_page_cost
- readable_external_table_timeout
- repl_catchup_within_range
- replication_timeout
- regex_flavor
- resource_cleanup_gangs_on_wait
- resource_select_only
- runaway_detector_activation_percent
- search_path
- seq_page_cost
- server_encoding
- server_version
- server_version_num
- shared_buffers
- shared_preload_libraries
- ssl
- ssl_ciphers
- standard_conforming_strings
- statement_mem
- statement_timeout
- stats_queue_level
- superuser_reserved_connections
- tcp_keepalives_count
- tcp_keepalives_idle
- tcp_keepalives_interval
- temp_buffers
- TimeZone
- timezone_abbreviations
- transaction_isolation
- transaction_read_only
- transform_null_equals
- unix_socket_directory
- unix_socket_group
- unix_socket_permissions
- update_process_title
- vacuum_cost_delay
- vacuum_cost_limit
- vacuum_cost_page_dirty
- vacuum_cost_page_hit
- vacuum_cost_page_miss
- vacuum_freeze_min_age
- validate_previous_free_tid
- vmem_process_interrupt
- writable_external_table_bufsize
- xid_stop_limit
- xid_warn_limit
- Summary of Built-in Functions
- Greenplum MapReduce Specification
- Greenplum PL/pgSQL Procedural Language
- Greenplum PostGIS Extension
- Greenplum PL/R Language Extension
- Greenplum PL/Python Language Extension
- Greenplum PL/Java Language Extension
- Greenplum MADlib Extension for Analytics
- Greenplum Partner Connector API
- Greenplum Fuzzy String Match Extension
- Summary of Greenplum Features
- UNIX Client Documentation
- Windows Client Documentation