本文详细介绍了如何在Linux环境下使用Greenplum的gpload工具远程加载数据到Greenplum数据库的过程。包括安装依赖软件、配置环境变量、启动gpfdist服务及通过gpload加载数据等步骤。
linux kettle greenplum 远程
两个参考文件中发现可以在etl服务器中加载gpload文件，通过文件将数据传送到远程的gp服务器中。
1、安装部署gpload以及相应的依赖文件
linux 操作系统：redhat 6.5
1-1）[root@localhost ~]# python
Python 2.6.6 (r266:84292, Sep 4 2013, 07:46:00)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-3)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> exit()
1-2）[root@localhost software]# tar -zxvf PyYAML-3.10.tar.gz
[root@localhost PyYAML-3.10]# python setup.py install
1-3）[root@localhost software]# tar -zxvf yaml-0.1.4.tar.gz
[root@localhost yaml-0.1.4]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... no
checking for cc... no
checking for cl.exe... no
configure: error: in `/software/yaml-0.1.4':
configure: error: no acceptable C compiler found in $PATH（缺少gcc依赖包
http://blog.csdn.net/duguduchong/article/details/8699774）
See `config.log' for more details
[root@localhost yaml-0.1.4]# make && make install
make: *** No targets specified and no makefile found. Stop.
导致第4步，第5步，第6步执行
如果第3步不报错，那么后续3步可以不能执行
1-4)yum --update
1-4-1、cp -R /media/RHEL_6.5\ x86_64\ Disc\ 1/ /softwa
re/bao
1-4-2、[root@localhost yum.repos.d]# cat rhel-source.repo
[oel6]
name = Enterprise Linux 6.0 DVD
baseurl=file:///software/bao/Server
gpgcheck=0
enabled=1
1-4-3、[root@localhost yum.repos.d]# yum update
Loaded plugins: product-id, refresh-packagekit, security, subscription-
: manager
This system is not registered to Red Hat Subscription Management. You can us
e subscription-manager to register.oel6 | 3.9 kB 00:00 ...
oel6/primary_db | 3.1 MB 00:00 ...
Setting up Update Process
No Packages marked for Update
1-4-4、yum install gcc
1-5）再次执行 [root@localhost yaml-0.1.4]# ./configure
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether gcc accepts -g... yes
checking for gcc option to accept ISO C89... none needed
checking for style of include used by make... GNU
checking dependency style of gcc... gcc3
checking how to run the C preprocessor... gcc -E
checking whether ln -s works... yes
checking whether make sets $(MAKE)... (cached) yes
checking build system type... x86_64-unknown-linux-gnu
checking host system type... x86_64-unknown-linux-gnu
checking for a sed that does not truncate output... /bin/sed
checking for grep that handles long lines and -e... /bin/grep
checking for egrep... /bin/grep -E
checking for fgrep... /bin/grep -F
checking for ld used by gcc... /usr/bin/ld
checking if the linker (/usr/bin/ld) is GNU ld... yes
checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B
checking the name lister (/usr/bin/nm -B) interface... BSD nm
checking the maximum length of command line arguments... 1966080
checking whether the shell understands some XSI constructs... yes
checking whether the shell understands "+="... yes
checking for /usr/bin/ld option to reload object files... -r
checking for objdump... objdump
checking how to recognize dependent libraries... pass_all
checking for ar... ar
checking for strip... strip
checking for ranlib... ranlib
checking command to parse /usr/bin/nm -B output from gcc object... ok
checking for ANSI C header files... yes
checking for sys/types.h... yes
checking for sys/stat.h... yes
checking for stdlib.h... yes
checking for string.h... yes
checking for memory.h... yes
checking for strings.h... yes
checking for inttypes.h... yes
checking for stdint.h... yes
checking for unistd.h... yes
checking for dlfcn.h... yes
checking for objdir... .libs
checking if gcc supports -fno-rtti -fno-exceptions... no
checking for gcc option to produce PIC... -fPIC -DPIC
checking if gcc PIC flag -fPIC -DPIC works... yes
checking if gcc static flag -static works... no
checking if gcc supports -c -o file.o... yes
checking if gcc supports -c -o file.o... (cached) yes
checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared
libraries... yeschecking whether -lc should be explicitly linked in... no
checking dynamic linker characteristics... GNU/Linux ld.so
checking how to hardcode library paths into programs... immediate
checking whether stripping libraries is possible... yes
checking if libtool supports shared libraries... yes
checking whether to build shared libraries... yes
checking whether to build static libraries... yes
checking for doxygen... false
checking for ANSI C header files... (cached) yes
checking for stdlib.h... (cached) yes
checking for an ANSI C-conforming const... yes
checking for size_t... yes
configure: creating ./config.status
config.status: creating yaml-0.1.pc
config.status: creating include/Makefile
config.status: creating src/Makefile
config.status: creating Makefile
config.status: creating tests/Makefile
config.status: creating win32/Makefile
config.status: creating config.h
config.status: executing depfiles commands
config.status: executing libtool commands
[root@localhost yaml-0.1.4]#
1-6）再次执行[root@localhost yaml-0.1.4]# make && make install
make all-recursive
make[1]: Entering directory `/software/yaml-0.1.4'
Making all in include
make[2]: Entering directory `/software/yaml-0.1.4/include'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/software/yaml-0.1.4/include'
Making all in src
make[2]: Entering directory `/software/yaml-0.1.4/src'
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT api.lo -MD -MP -MF .deps/api.Tpo -c -o api.lo api.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT api.
lo -MD -MP -MF .deps/api.Tpo -c api.c -fPIC -DPIC -o .libs/api.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT api.
lo -MD -MP -MF .deps/api.Tpo -c api.c -o api.o >/dev/null 2>&1mv -f .deps/api.Tpo .deps/api.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT reader.lo -MD -MP -MF .deps/reader.Tpo -c -o reader.lo reader.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT read
er.lo -MD -MP -MF .deps/reader.Tpo -c reader.c -fPIC -DPIC -o .libs/reader.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT read
er.lo -MD -MP -MF .deps/reader.Tpo -c reader.c -o reader.o >/dev/null 2>&1mv -f .deps/reader.Tpo .deps/reader.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT scanner.lo -MD -MP -MF .deps/scanner.Tpo -c -o scanner.lo scanner.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT scan
ner.lo -MD -MP -MF .deps/scanner.Tpo -c scanner.c -fPIC -DPIC -o .libs/scanner.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT scan
ner.lo -MD -MP -MF .deps/scanner.Tpo -c scanner.c -o scanner.o >/dev/null 2>&1mv -f .deps/scanner.Tpo .deps/scanner.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT parser.lo -MD -MP -MF .deps/parser.Tpo -c -o parser.lo parser.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT pars
er.lo -MD -MP -MF .deps/parser.Tpo -c parser.c -fPIC -DPIC -o .libs/parser.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT pars
er.lo -MD -MP -MF .deps/parser.Tpo -c parser.c -o parser.o >/dev/null 2>&1mv -f .deps/parser.Tpo .deps/parser.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT loader.lo -MD -MP -MF .deps/loader.Tpo -c -o loader.lo loader.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT load
er.lo -MD -MP -MF .deps/loader.Tpo -c loader.c -fPIC -DPIC -o .libs/loader.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT load
er.lo -MD -MP -MF .deps/loader.Tpo -c loader.c -o loader.o >/dev/null 2>&1mv -f .deps/loader.Tpo .deps/loader.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT writer.lo -MD -MP -MF .deps/writer.Tpo -c -o writer.lo writer.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT writ
er.lo -MD -MP -MF .deps/writer.Tpo -c writer.c -fPIC -DPIC -o .libs/writer.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT writ
er.lo -MD -MP -MF .deps/writer.Tpo -c writer.c -o writer.o >/dev/null 2>&1mv -f .deps/writer.Tpo .deps/writer.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT emitter.lo -MD -MP -MF .deps/emitter.Tpo -c -o emitter.lo emitter.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT emit
ter.lo -MD -MP -MF .deps/emitter.Tpo -c emitter.c -fPIC -DPIC -o .libs/emitter.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT emit
ter.lo -MD -MP -MF .deps/emitter.Tpo -c emitter.c -o emitter.o >/dev/null 2>&1mv -f .deps/emitter.Tpo .deps/emitter.Plo
/bin/sh ../libtool --tag=CC --mode=compile gcc -DHAVE_CONFIG_H -I. -I.. -
I../include -g -O2 -MT dumper.lo -MD -MP -MF .deps/dumper.Tpo -c -o dumper.lo dumper.clibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT dump
er.lo -MD -MP -MF .deps/dumper.Tpo -c dumper.c -fPIC -DPIC -o .libs/dumper.olibtool: compile: gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT dump
er.lo -MD -MP -MF .deps/dumper.Tpo -c dumper.c -o dumper.o >/dev/null 2>&1mv -f .deps/dumper.Tpo .deps/dumper.Plo
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -release 0 -version-in
fo 2:2:0 -o libyaml.la -rpath /usr/local/lib api.lo reader.lo scanner.lo parser.lo loader.lo writer.lo emitter.lo dumper.lo libtool: link: gcc -shared .libs/api.o .libs/reader.o .libs/scanner.o .libs
/parser.o .libs/loader.o .libs/writer.o .libs/emitter.o .libs/dumper.o -Wl,-soname -Wl,libyaml-0.so.2 -o .libs/libyaml-0.so.2.0.2libtool: link: (cd ".libs" && rm -f "libyaml-0.so.2" && ln -s "libyaml-0.so.
2.0.2" "libyaml-0.so.2")libtool: link: (cd ".libs" && rm -f "libyaml.so" && ln -s "libyaml-0.so.2.0.
2" "libyaml.so")libtool: link: ar cru .libs/libyaml.a api.o reader.o scanner.o parser.o loa
der.o writer.o emitter.o dumper.olibtool: link: ranlib .libs/libyaml.a
libtool: link: ( cd ".libs" && rm -f "libyaml.la" && ln -s "../libyaml.la" "
libyaml.la" )make[2]: Leaving directory `/software/yaml-0.1.4/src'
Making all in .
make[2]: Entering directory `/software/yaml-0.1.4'
make[2]: Leaving directory `/software/yaml-0.1.4'
Making all in tests
make[2]: Entering directory `/software/yaml-0.1.4/tests'
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT run-scanner.o -MD -M
P -MF .deps/run-scanner.Tpo -c -o run-scanner.o run-scanner.cmv -f .deps/run-scanner.Tpo .deps/run-scanner.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o run-scanner run-s
canner.o ../src/libyaml.la libtool: link: gcc -g -O2 -o .libs/run-scanner run-scanner.o ../src/.libs/l
ibyaml.so -Wl,-rpath -Wl,/usr/local/libgcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT run-parser.o -MD -MP
-MF .deps/run-parser.Tpo -c -o run-parser.o run-parser.cmv -f .deps/run-parser.Tpo .deps/run-parser.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o run-parser run-pa
rser.o ../src/libyaml.la libtool: link: gcc -g -O2 -o .libs/run-parser run-parser.o ../src/.libs/lib
yaml.so -Wl,-rpath -Wl,/usr/local/libgcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT run-loader.o -MD -MP
-MF .deps/run-loader.Tpo -c -o run-loader.o run-loader.cmv -f .deps/run-loader.Tpo .deps/run-loader.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o run-loader run-lo
ader.o ../src/libyaml.la libtool: link: gcc -g -O2 -o .libs/run-loader run-loader.o ../src/.libs/lib
yaml.so -Wl,-rpath -Wl,/usr/local/libgcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT run-emitter.o -MD -M
P -MF .deps/run-emitter.Tpo -c -o run-emitter.o run-emitter.cmv -f .deps/run-emitter.Tpo .deps/run-emitter.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o run-emitter run-emitter.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/run-emitter run-emitter.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT run-dumper.o -MD -MP -MF .deps/run-dumper.Tpo -c -o run-dumper.o run-dumper.c
mv -f .deps/run-dumper.Tpo .deps/run-dumper.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o run-dumper run-dumper.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/run-dumper run-dumper.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT example-reformatter.o -MD -MP -MF .deps/example-reformatter.Tpo -c -o example-reformatter.o example-reformatter.c
mv -f .deps/example-reformatter.Tpo .deps/example-reformatter.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o example-reformatter example-reformatter.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/example-reformatter example-reformatter.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT example-reformatter-alt.o -MD -MP -MF .deps/example-reformatter-alt.Tpo -c -o example-reformatter-alt.o example-reformatter-alt.c
mv -f .deps/example-reformatter-alt.Tpo .deps/example-reformatter-alt.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o example-reformatter-alt example-reformatter-alt.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/example-reformatter-alt example-reformatter-alt.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT example-deconstructor.o -MD -MP -MF .deps/example-deconstructor.Tpo -c -o example-deconstructor.o example-deconstructor.c
mv -f .deps/example-deconstructor.Tpo .deps/example-deconstructor.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o example-deconstructor example-deconstructor.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/example-deconstructor example-deconstructor.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
gcc -DHAVE_CONFIG_H -I. -I.. -I../include -g -O2 -MT example-deconstructor-alt.o -MD -MP -MF .deps/example-deconstructor-alt.Tpo -c -o example-deconstructor-alt.o example-deconstructor-alt.c
mv -f .deps/example-deconstructor-alt.Tpo .deps/example-deconstructor-alt.Po
/bin/sh ../libtool --tag=CC --mode=link gcc -g -O2 -o example-deconstructor-alt example-deconstructor-alt.o ../src/libyaml.la
libtool: link: gcc -g -O2 -o .libs/example-deconstructor-alt example-deconstructor-alt.o ../src/.libs/libyaml.so -Wl,-rpath -Wl,/usr/local/lib
make[2]: Leaving directory `/software/yaml-0.1.4/tests'
Making all in win32
make[2]: Entering directory `/software/yaml-0.1.4/win32'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/software/yaml-0.1.4/win32'
make[1]: Leaving directory `/software/yaml-0.1.4'
Making install in include
make[1]: Entering directory `/software/yaml-0.1.4/include'
make[2]: Entering directory `/software/yaml-0.1.4/include'
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/include" || /bin/mkdir -p "/usr/local/include"
/usr/bin/install -c -m 644 yaml.h '/usr/local/include/.'
make[2]: Leaving directory `/software/yaml-0.1.4/include'
make[1]: Leaving directory `/software/yaml-0.1.4/include'
Making install in src
make[1]: Entering directory `/software/yaml-0.1.4/src'
make[2]: Entering directory `/software/yaml-0.1.4/src'
test -z "/usr/local/lib" || /bin/mkdir -p "/usr/local/lib"
/bin/sh ../libtool --mode=install /usr/bin/install -c libyaml.la '/usr/local/lib'
libtool: install: /usr/bin/install -c .libs/libyaml-0.so.2.0.2 /usr/local/lib/libyaml-0.so.2.0.2
libtool: install: (cd /usr/local/lib && { ln -s -f libyaml-0.so.2.0.2 libyaml-0.so.2 || { rm -f libyaml-0.so.2 && ln -s libyaml-0.so.2.0.2 libyaml-0.so.2; }; })
libtool: install: (cd /usr/local/lib && { ln -s -f libyaml-0.so.2.0.2 libyaml.so || { rm -f libyaml.so && ln -s libyaml-0.so.2.0.2 libyaml.so; }; })
libtool: install: /usr/bin/install -c .libs/libyaml.lai /usr/local/lib/libyaml.la
libtool: install: /usr/bin/install -c .libs/libyaml.a /usr/local/lib/libyaml.a
libtool: install: chmod 644 /usr/local/lib/libyaml.a
libtool: install: ranlib /usr/local/lib/libyaml.a
libtool: finish: PATH="/usr/lib64/qt-3.3/bin:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin:/sbin" ldconfig -n /usr/local/lib
----------------------------------------------------------------------
Libraries have been installed in:
/usr/local/lib
If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
- add LIBDIR to the `LD_LIBRARY_PATH' environment variable
during execution
- add LIBDIR to the `LD_RUN_PATH' environment variable
during linking
- use the `-Wl,-rpath -Wl,LIBDIR' linker flag
- have your system administrator add LIBDIR to `/etc/ld.so.conf'
See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/software/yaml-0.1.4/src'
make[1]: Leaving directory `/software/yaml-0.1.4/src'
Making install in .
make[1]: Entering directory `/software/yaml-0.1.4'
make[2]: Entering directory `/software/yaml-0.1.4'
make[2]: Nothing to be done for `install-exec-am'.
test -z "/usr/local/lib/pkgconfig" || /bin/mkdir -p "/usr/local/lib/pkgconfig"
/usr/bin/install -c -m 644 yaml-0.1.pc '/usr/local/lib/pkgconfig'
make[2]: Leaving directory `/software/yaml-0.1.4'
make[1]: Leaving directory `/software/yaml-0.1.4'
Making install in tests
make[1]: Entering directory `/software/yaml-0.1.4/tests'
make[2]: Entering directory `/software/yaml-0.1.4/tests'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/software/yaml-0.1.4/tests'
make[1]: Leaving directory `/software/yaml-0.1.4/tests'
Making install in win32
make[1]: Entering directory `/software/yaml-0.1.4/win32'
make[2]: Entering directory `/software/yaml-0.1.4/win32'
make[2]: Nothing to be done for `install-exec-am'.
make[2]: Nothing to be done for `install-data-am'.
make[2]: Leaving directory `/software/yaml-0.1.4/win32'
make[1]: Leaving directory `/software/yaml-0.1.4/win32'
[root@localhost yaml-0.1.4]#
1-7)[root@localhost software]# unzip greenplum-loaders-4.3.8.1-build-1-RHEL5-x86_64.zip
Archive: greenplum-loaders-4.3.8.1-build-1-RHEL5-x86_64.zip
inflating: greenplum-loaders-4.3.8.1-build-1-RHEL5-x86_64.bin
1-8)[root@localhost software]# ./greenplum-loaders-4.3.8.1-build-1-RHEL5-x86_64.bin
....
*******************************************************************************
Install Greenplum Loaders into </usr/local/greenplum-loaders-4.3.8.1-build-1>? [yes | no]
********************************************************************************
yes
********************************************************************************
/usr/local/greenplum-loaders-4.3.8.1-build-1 does not exist.
Create /usr/local/greenplum-loaders-4.3.8.1-build-1 ? [ yes | no ]
(Selecting no will exit the installer)
********************************************************************************
yes
Extracting product to /usr/local/greenplum-loaders-4.3.8.1-build-1
********************************************************************************
Installation complete.
Greenplum Loaders is installed in:
/usr/local/greenplum-loaders-4.3.8.1-build-1
Pivotal Greenplum documentation is available
for download at http://docs.gopivotal.com/gpdb
1-9）[root@localhost greenplum-loaders-4.3.8.1-build-1]# cat greenplum_loaders_pa
th.sh if [ `uname -s` = "HP-UX" ]; then
set +o nounset
fi
GPHOME_LOADERS=/usr/local/greenplum-loaders-4.3.8.1-build-1
PATH=${GPHOME_LOADERS}/bin:${GPHOME_LOADERS}/ext/python/bin:${PATH}
PYTHONPATH=${GPHOME_LOADERS}/bin/ext:${PYTHONPATH}
export GPHOME_LOADERS
export PATH
export PYTHONPATH
# Mac OSX uses a different library path variable
if [ xDarwin = x`uname -s` ]; then
DYLD_LIBRARY_PATH=${GPHOME_LOADERS}/lib:${GPHOME_LOADERS}/ext/python/lib:$
{DYLD_LIBRARY_PATH} export DYLD_LIBRARY_PATH
else
LD_LIBRARY_PATH=${GPHOME_LOADERS}/lib:${GPHOME_LOADERS}/ext/python/lib:${L
D_LIBRARY_PATH} export LD_LIBRARY_PATH
fi
# AIX uses yet another library path variable
# Also, Python on AIX requires special copies of some libraries. Hence, lib
/pware.if [ xAIX = x`uname -s` ]; then
LIBPATH=${GPHOME_LOADERS}/lib/pware:${GPHOME_LOADERS}/lib:${GPHOME_LOADERS
}/ext/python/lib:/usr/lib/threads:${LIBPATH} export LIBPATH
GP_LIBPATH_FOR_PYTHON=${GPHOME_LOADERS}/lib/pware
export GP_LIBPATH_FOR_PYTHON
fi
if [ "$1" != "-q" ]; then
type python >/dev/null 2>&1
if [ $? -ne 0 ]; then
echo "Warning: Python not found. Python-2.5.1 or better is required to
run gpload." fi
fi
[root@localhost greenplum-loaders-4.3.8.1-build-1]#
如果第8步中都默认yes,那么不需要修改文件
1-10)[root@localhost ~]# vi .bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin
export PATH
#首先你的数据库，地址，端口，用户
export PGDATABASE=tutorial
export PGHOST=master
export PGPORT=5432
export PGUSER=user1
export PGPASSWORD=user1
1-11)[root@localhost ~]# source ~/.bash_profile
2、启动gpfdsit
1）[root@localhost bin]# nohup /usr/local/greenplum-loaders-4.3.8.1-build-1/bin/gpfdist -d /home/admin/ -p 8888 > /tmp/gpfdist.log 2>&1 &
[1] 5474
[root@localhost bin]# cat /tmp/gpfdist.log
nohup: ignoring input
/usr/local/greenplum-loaders-4.3.8.1-build-1/bin/gpfdist: error while loading shared libraries: libyaml-0.so.1: cannot open shared object file: No such file or directory（《
https://discuss.pivotal.io/hc/zh-cn/community/posts/206424348-Error-Message-gpfdist-error-while-loading-shared-libraries-libyaml-0-so-1》）
2）执行[root@localhost greenplum-loaders-4.3.8.1-build-1]# source greenplum_loaders_path.sh
3）再次执行
[root@localhost greenplum-loaders-4.3.8.1-build-1]# nohup /usr/local/greenpl
um-loaders-4.3.8.1-build-1/bin/gpfdist -d /home/admin/ -p 8888 > /tmp/gpfdist.log 2>&1 &[1] 5535
[root@localhost greenplum-loaders-4.3.8.1-build-1]# cat /tmp/gpfdist.log
nohup: ignoring input
2016-12-14 23:57:28 5535 INFO Before opening listening sockets - following l
istening sockets are available:2016-12-14 23:57:28 5535 INFO IPV6 socket: [::]:8888
2016-12-14 23:57:28 5535 INFO IPV4 socket: 0.0.0.0:8888
2016-12-14 23:57:28 5535 INFO Trying to open listening socket:
2016-12-14 23:57:28 5535 INFO IPV6 socket: [::]:8888
2016-12-14 23:57:28 5535 INFO Opening listening socket succeeded
2016-12-14 23:57:28 5535 INFO Trying to open listening socket:
2016-12-14 23:57:28 5535 INFO IPV4 socket: 0.0.0.0:8888
Serving HTTP on port 8888, directory /home/admin
3、加载数据
[root@kettle bin]# gpload -f my_load.yml
2016-12-15 04:05:15|INFO|gpload session started 2016-12-15 04:05:15
2016-12-15 04:05:15|INFO|started gpfdist -p 8888 -P 8889 -f "/home/admin/gpe
xtdata/c.txt" -t 302016-12-15 04:05:17|ERROR|ERROR: connection with gpfdist failed for gpfdist
://192.168.1.101:8889//home/admin/gpextdata/c.txt. effective url: http://192.168.1.101:8889//home/admin/gpextdata/c.txt. error code = 113 (No route to host（
http://blog.sina.com.cn/s/blog_9151e7300101ksui.html）) (seg0 slice1 slave1:40000 pid=2566) encountered while running INSERT INTO "public"."test001" ("id","name") SELE
CT "id","name" FROM ext_gpload_97a037fe_c2a5_11e6_bb0b_08002741af212016-12-15 04:05:17|INFO|rows Inserted = 0
2016-12-15 04:05:17|INFO|rows Updated = 0
2016-12-15 04:05:17|INFO|data formatting errors = 0
2016-12-15 04:05:17|INFO|gpload failed
--关闭防火墙
[root@kettle bin]# gpload -f /home/admin/gpextdata/abc.yml
2016-12-15 04:34:29|INFO|gpload session started 2016-12-15 04:34:29
2016-12-15 04:34:29|INFO|started gpfdist -p 8877 -P 8878 -f "/home/admin/gpextdata/c.txt" -t 302016-12-15 04:34:30|INFO|running time: 0.31 seconds
2016-12-15 04:34:30|INFO|rows Inserted = 2
2016-12-15 04:34:30|INFO|rows Updated = 0
2016-12-15 04:34:30|INFO|data formatting errors = 0
2016-12-15 04:34:30|INFO|gpload succeeded
[root@kettle bin]#
[root@kettle bin]# cat abc.yml
VERSION: 1.0.0.1
DATABASE: tutorial
USER: user1
HOST: 192.168.1.80
PORT: 5432
GPLOAD:
INPUT:
- SOURCE:
LOCAL_HOSTNAME:
- 192.168.1.101
PORT: 8888
FILE: ['/home/admin/gpextdata/c.txt']
- COLUMNS:
- id:
- "name":
- FORMAT: TEXT
- DELIMITER: ','
- QUOTE: ''
- HEADER: FALSE
- ENCODING: UTF8
- ERROR_LIMIT: 50
OUTPUT:
- TABLE: "public.test001"
- MODE: insert
4、kettle调用
2016/12/15 05:00:25 - Text file input.0 - Opening file: file:///home/admin/gpextdata/b.txt
2016/12/15 05:00:25 - Text file input.0 - Finished processing (I=7, O=0, R=0, W=7, U=1, E=0)
2016/12/15 05:00:25 - Greenplum Load.0 - Executing: /usr/local/greenplum-loaders-4.3.8.1-build-1/bin/gpload -f /home/admin/gpextdata/my_flag.yml
2016/12/15 05:00:25 - Greenplum Load.0 - Exit Value of psql: 127
2016/12/15 05:00:25 - Greenplum Load.0 - ERROR (version 5.4.0.1-130, build 1 from 2015-06-14_12-34-55 by buildguy) : Error in step, asking everyone to stop because of:
2016/12/15 05:00:25 - Greenplum Load.0 - Exit Value of psql: 127
2016/12/15 05:00:25 - Greenplum Load.0 - Finished processing (I=0, O=7, R=7, W=7, U=0, E=1)
2016/12/15 05:00:25 - Greenplum Load.0 - ERROR>/usr/local/greenplum-loaders-4.3.8.1-build-1/bin/gpload: line 14: exec: gpload.py: not found
2016/12/15 05:00:25 - greepload - ERROR (version 5.4.0.1-130, build 1 from 2015-06-14_12-34-55 by buildguy) : Errors detected!
2016/12/15 05:00:25 - Spoon - The transformation has finished!!
现在到了这一步更不能放弃
1）将主机的bin目录其他文件拷贝过来，重新执行报错
2）去主机中执行exec gpload.py
[gpadmin@master bin]$ exec gpload.py
COMMAND NAME: gpload
Runs a load job as defined in a YAML formatted control file.
*****************************************************
SYNOPSIS
*****************************************************
gpload -f <control_file> [-l <log_file>]
[-h <hostname>] [-p <port>] [-U <username>] [-d <database>] [-W]
[--gpfdist_timeout <seconds>] [[-v | -V] | [-q]] [-D]
gpload -?
gpload --version
*****************************************************
PREREQUISITES
*****************************************************
The client machine where gpload is executed must have the following:
* Python 2.6.2 or later, pygresql (the Python interface to PostgreSQL),
and pyyaml. Note that Python and the required Python libraries
are included with the Greenplum Database server installation, so if
you have Greenplum Database installed on the machine where gpload
is running, you do not need a separate Python installation.
Note: Greenplum Loaders for Windows supports only Python 2.5
(available from www.python.org).
* The gpfdist parallel file distribution program installed and in
your $PATH. This program is located in $GPHOME/bin of your
Greenplum Database server installation.
* Network access to and from all hosts in your Greenplum Database
array (master and segments).
* Network access to and from the hosts where the data to be loaded
resides (ETL servers).
*****************************************************
DESCRIPTION
*****************************************************
gpload is a data loading utility that acts as an interface to
Greenplum Database's parallel loading feature.
Using a load specification defined in a YAML formatted control file,
gpload executes a load by invoking the Greenplum parallel file
server (gpfdist), creating an external table definition based on
the source data defined, and executing an INSERT, UPDATE or MERGE
operation to load the source data into the target table in the
database.
*****************************************************
OPTIONS
*****************************************************
...
*****************************************************
SEE ALSO
*****************************************************
gpfdist, CREATE EXTERNAL TABLE
ERROR: configuration file required
[root@master ~]#
那么就是某些变量没有配置或者没有配对
重新查看官网文档配置环境路径
Configuring the Command-Line Load Tools
As a convenience, a greenplum_loaders_path.sh file is provided in your load tools installation directory following installation. It has the following environment variable settings:
- GREENPLUM_LOADERS_HOME — The installation directory of the Greenplum Database load tools.
- PATH — The path to the data loading command-line utilities.
- LD_LIBRARY_PATH — The path to additional Python library files needed for gpload.
You can source this file in your user's startup shell profile (such as .bashrc or .bash_profile).
For example, you could add a line similar to the following to your chosen profile files (making sure the right install path is used):
source greenplum-db-4.3.x.x/greenplum_loaders_path.sh
After editing the chosen profile file, source it as the correct user to make the changes active. For example:
source ~/.bashrc
之前没有配置字符集，一直在报错：
Exit Value of psql: 127,
其实这个报错网上查找资料，修改环境变量最后都没有成功，直到配置了字符集
其实这个报错网上查找资料，修改环境变量最后都没有成功，直到配置了字符集
偶然将下图中的字符集选择后，就可以调用，
问题来了：
kettle 调用过程中需要字符集，为什么直接调用不需要字符集
问题来了：
kettle 调用过程中需要字符集，为什么直接调用不需要字符集
2016/12/16 01:28:49 - Spoon - Transformation opened.
2016/12/16 01:28:49 - Spoon - Launching transformation [greepload]...
2016/12/16 01:28:49 - Spoon - Started the transformation execution.
2016/12/16 01:28:49 - greepload - Dispatching started for transformation [greepload]
2016/12/16 01:28:49 - Text file input.0 - Opening file: file:///home/admin/gpextdata/b.txt
2016/12/16 01:28:49 - Text file input.0 - Finished processing (I=4, O=0, R=0, W=4, U=1, E=0)
2016/12/16 01:28:49 - Greenplum Load.0 - Executing: /usr/local/greenplum-loaders-4.3.8.1-build-1/bin/gpload -f /home/admin/gpextdata/my_flag.yml
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|gpload session started 2016-12-16 01:28:50
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|started gpfdist -p 8899 -P 8900 -f "/home/admin/gpextdata/c.txt" -t 30
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|running time: 0.49 seconds
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|rows Inserted = 3
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|rows Updated = 1
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|data formatting errors = 0
2016/12/16 01:28:50 - Greenplum Load.0 - OUTPUT>2016-12-16 01:28:50|INFO|gpload succeeded
2016/12/16 01:28:50 - Greenplum Load.0 - Exit Value of psql: 0
2016/12/16 01:28:50 - Greenplum Load.0 - Finished processing (I=0, O=4, R=4, W=4, U=0, E=0)
2016/12/16 01:28:50 - Spoon - The transformation has finished!!
详情配置请查看另外一篇文档
http://blog.itpub.net/29162273/viewspace-2127884/
来自 “ ITPUB博客 ” ，链接：http://blog.itpub.net/29162273/viewspace-2130770/，如需转载，请注明出处，否则将追究法律责任。
转载于:http://blog.itpub.net/29162273/viewspace-2130770/
扫一扫
716
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
