ambar2.7.3 安裝 impala

1.下載ambari-impala-service

VERSION=`hdp-select status hadoop-client | sed 's/hadoop-client - \([0-9]\.[0-9]\).*/\1/'`
sudo git clone https://github.com/cas-bigdatalab/ambari-impala-service.git /var/lib/ambari-server/resources/stacks/HDP/$VERSION/services/IMPALA

2./etc/yum.repos.d新建impala.repo

[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat	or CentOS 6 x86_64
name=Cloudera's Distribution for Hadoop, Version 5
baseurl=https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/5.8.0
gpgkey =https://archive.cloudera.com/cdh5/redhat/7/x86_64/cdh/RPM-GPG-KEY-cloudera
gpgcheck = 1

3.重啓ambari-server

sudo service ambari-server restart

4.安裝impala

Actions-->Add service -->impala，然後一步一步安裝

 

中間發生一次報錯信息：

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 38, in <module>
    BeforeAnyHook().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py", line 31, in hook
    setup_users()
  File "/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/shared_initialization.py", line 50, in setup_users
    groups = params.user_to_groups_dict[user],
KeyError: u'impala'
Error: Error: Unable to run the custom hook script ['/usr/bin/python', '/var/lib/ambari-agent/cache/stack-hooks/before-ANY/scripts/hook.py', 'ANY', '/var/lib/ambari-agent/data/command-863.json', '/

1 命令行

# cd /var/lib/ambari-server/resources/scripts
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a get -c cluster-env |grep -i ignore_groupsusers_create
"ignore_groupsusers_create": "false",
# python configs.py -u admin -p admin -n $cluster_name -l $ambari_server -t 8080 -a set -c cluster-env -k ignore_groupsusers_create -v true

詳細參數詳見

# python configs.py --help

2 rest api

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env"

從返回的items中找到tag，然後再請求（增加tag參數）：

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X GET "http://ambari.server:8080/api/v1/clusters/$cluster_name/configurations?type=cluster-env&tag=$tag"

返回結果中包含所有的property

"properties" : {
"agent_mounts_ignore_list" : "",
"alerts_repeat_tolerance" : "1",
"enable_external_ranger" : "false",
"fetch_nonlocal_groups" : "true",
"hide_yarn_memory_widget" : "false",
"ignore_bad_mounts" : "false",
"ignore_groupsusers_create" : "false",

修改需要先將上面的配置保存到一個文件，比如/tmp/configurations.tmp，然後修改，然後再調用接口:

curl --user admin:admin -i -H ‘X-Requested-By: ambari‘ -X PUT -d @/tmp/configurations.tmp http://ambari.server:8080/api/v1/clusters/$cluster_name

然後再安裝即可；

因爲要去cloudera官網下載impala的rpm文件，安裝過程比較慢，可能會失敗，多試幾次就好了

如果需要重裝 報錯

Traceback (most recent call last):
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala-catalog.py", line 36, in <module>
    ImpalaCatalog().execute()
  File "/usr/lib/ambari-agent/lib/resource_management/libraries/script/script.py", line 352, in execute
    method(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala-catalog.py", line 12, in install
    self.installImpala(env)
  File "/var/lib/ambari-agent/cache/stacks/HDP/3.1/services/IMPALA2.6/package/scripts/impala_base.py", line 19, in installImpala
    Package(pack)
  File "/usr/lib/ambari-agent/lib/resource_management/core/base.py", line 166, in __init__
    self.env.run()
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 160, in run
    self.run_action(resource, action)
  File "/usr/lib/ambari-agent/lib/resource_management/core/environment.py", line 124, in run_action
    provider_action()
  File "/usr/lib/ambari-agent/lib/resource_management/core/providers/packaging.py", line 30, in action_install
    self._pkg_manager.install_package(package_name, self.__create_context())
  File "/usr/lib/ambari-agent/lib/ambari_commons/repo_manager/yum_manager.py", line 219, in install_package
    shell.repository_manager_executor(cmd, self.properties, context)
  File "/usr/lib/ambari-agent/lib/ambari_commons/shell.py", line 753, in repository_manager_executor
    raise RuntimeError(message)
RuntimeError: Failed to execute command '/usr/bin/yum -y install impala-server', exited with code '1', message: 'https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/repodata/repomd.xml: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

To address this issue please refer to the below knowledge base article 



https://access.redhat.com/articles/1320623



If above article doesn't help to resolve this issue please create a bug on https://bugs.centos.org/



https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/bigtop-utils-0.7.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/avro-libs-1.7.6%2Bcdh5.16.1%2B143-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/x86_64/impala-2.12.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/x86_64/impala-server-2.12.0%2Bcdh5.16.1%2B0-1.cdh5.16.1.p0.3.el7.x86_64.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/parquet-1.5.0%2Bcdh5.16.1%2B200-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/parquet-format-2.1.0%2Bcdh5.16.1%2B22-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/sentry-1.5.1%2Bcdh5.16.1%2B559-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.

https://cloudera.proxy.ustclug.org/cdh5/redhat/7/x86_64/cdh/5.8/RPMS/x86_64/RPMS/noarch/solr-4.10.3%2Bcdh5.16.1%2B532-1.cdh5.16.1.p0.3.el7.noarch.rpm: [Errno 14] HTTPS Error 404 - Not Found

Trying other mirror.





Error downloading packages:

  avro-libs-1.7.6+cdh5.16.1+143-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  impala-2.12.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.x86_64: [Errno 256] No more mirrors to try.

  parquet-format-2.1.0+cdh5.16.1+22-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  sentry-1.5.1+cdh5.16.1+559-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  solr-4.10.3+cdh5.16.1+532-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  parquet-1.5.0+cdh5.16.1+200-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

  impala-server-2.12.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.x86_64: [Errno 256] No more mirrors to try.

  bigtop-utils-0.7.0+cdh5.16.1+0-1.cdh5.16.1.p0.3.el7.noarch: [Errno 256] No more mirrors to try.

需要執行

yum clean all
yum list update
yum makecache
yum repolist

5.hdfs增加配置

#HDFS config we need add below config to /etc/hadoop/conf/core-site.xml
<property>
    <name>dfs.client.read.shortcircuit</name> 
   <value>true</value>
</property>

<property>
    <name>dfs.client.read.shortcircuit.skip.checksum</name>
        <value>false</value>
</property>

<property> 
    <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> 
    <value>true</value>
</property>

we need add below config to /etc/hadoop/conf/hdfs-site.xml
<property>
    <name>dfs.datanode.hdfs-blocks-metadata.enabled</name> 
    <value>true</value>
</property>
<property> 
    <name>dfs.block.local-path-access.user</name> 
    <value>impala</value>
</property>
<property>
    <name>dfs.client.file-block-storage-locations.timeout.millis</name>
    <value>60000</value>
</property>

 6.重啓hdfs

7.啓動impala

報錯：

cat impala-server.log （每臺impala機器都要做）
Unable to find Java. JAVA_HOME should be set in /etc/default/bigtop-utils

vim /etc/default/bigtop-utils

JAVA_HOME=/opt/jdk1.8.0_201 （與/etc/profile保持一樣）

報錯

cat /var/log/impala/impalad.ERROR

hdfsBuilderConnect(forceNewInstance=0, nn=default, port=0, kerbTicketCachePath=(NULL), userName=(NULL)) error:
ClassNotFoundException: com.ctc.wstx.io.InputBootstrapperjava.lang.NoClassDefFoundError: com/ctc/wstx/io/InputBootstrapper
Caused by: java.lang.ClassNotFoundException: com.ctc.wstx.io.InputBootstrapper
    at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
Wrote minidump to /var/log/impala/minidumps/impalad/0a803f89-a804-4bb4-242e02b9-94eba9b1.dmp

把hbase的jar包拷貝到 /usr/lib/impala/lib 並創建軟連接

這裏注意根據安裝的impala的版本對應的cdh的版本選擇對應的hbase的版本，否則可能出現版本不兼容的問題

ln -s hbase-annotations-1.2.0-cdh5.11.0.jar hbase-annotations.jar
ln -s hbase-client-1.2.0-cdh5.11.0.jar hbase-client.jar
ln -s hbase-common-1.2.0-cdh5.11.0.jar hbase-common.jar
ln -s hbase-examples-1.2.0-cdh5.11.0.jar hbase-examples.jar
ln -s hbase-external-blockcache-1.2.0-cdh5.11.0.jar hbase-external-blockcache.jar
ln -s hbase-hadoop2-compat-1.2.0-cdh5.11.0.jar hbase-hadoop2-compat.jar
ln -s hbase-hadoop-compat-1.2.0-cdh5.11.0.jar hbase-hadoop-compat.jar
ln -s hbase-it-1.2.0-cdh5.11.0.jar hbase-it.jar
ln -s hbase-prefix-tree-1.2.0-cdh5.11.0.jar hbase-prefix-tree.jar
ln -s hbase-procedure-1.2.0-cdh5.11.0.jar hbase-procedure.jar
ln -s hbase-protocol-1.2.0-cdh5.11.0.jar hbase-protocol.jar
ln -s hbase-resource-bundle-1.2.0-cdh5.11.0.jar hbase-resource-bundle.jar
ln -s hbase-rest-1.2.0-cdh5.11.0.jar hbase-rest.jar
ln -s hbase-rsgroup-1.2.0-cdh5.11.0.jar hbase-rsgroup.jar
ln -s hbase-server-1.2.0-cdh5.11.0.jar hbase-server.jar
ln -s hbase-shell-1.2.0-cdh5.11.0.jar hbase-shell.jar
ln -s hbase-spark-1.2.0-cdh5.11.0.jar hbase-spark.jar
ln -s hbase-thrift-1.2.0-cdh5.11.0.jar hbase-thrift.jar

報錯

E0524 17:18:04.260188  1741 impala-server.cc:279] Invalid short-circuit reads co
  - Impala cannot read or execute the parent directory of dfs.domain.socket.path

在配置文件中查看dfs.domain.socket.path 爲$path

chmod 775  ../$path

def:

http://www.mamicode.com/info-detail-2589065.html

https://github.com/cas-bigdatalab/ambari-impala-service/(ambari-impala github官網有詳細步驟持續更新)