cdh6.3.2安裝apache-kylin-3.0.2-bin-cdh60

首先,由於本人是cdh6.3.2,故點擊下載選擇下圖cdh60包

[root@cdh632-worker03 ~]# cd /opt/soft/
[root@cdh632-worker03 soft]# tar zxf apache-kylin-3.0.2-bin-cdh60.tar.gz -C /opt/module/
[root@cdh632-worker03 soft]# cd /opt/module/apache-kylin-3.0.2-bin-cdh60/
爲解決報錯01:
[root@cdh632-worker03 ~]# vi /opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/bin/hbase
# CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar
# 將上面的註釋掉修改爲如下
CLASSPATH=${CLASSPATH}:$JAVA_HOME/lib/tools.jar:/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/lib/*
[root@cdh632-worker03 apache-kylin-3.0.2-bin-cdh60]# vi ~/.bashrc
# kylin3.0.2
# 爲解決報錯02
export HADOOP_USER_NAME=hdfs
# 爲解決報錯03
export SPARK_HOME=/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark
export KYLIN_HOME=/opt/module/apache-kylin-3.0.2-bin-cdh60
[root@cdh632-worker03 apache-kylin-3.0.2-bin-cdh60]# $KYLIN_HOME/bin/check-env.sh
Retrieving hadoop conf dir...
報錯01:
Error: Could not find or load main class org.apache.hadoop.hbase.util.GetJavaProperty
KYLIN_HOME is set to /opt/module/apache-kylin-3.0.2-bin-cdh60
報錯02:
mkdir: Permission denied: user=root, access=WRITE, inode="/":hdfs:supergroup:drwxr-xr-x
Failed to create /kylin. Please make sure the user has right to access /kylin
[root@cdh632-worker03 apache-kylin-3.0.2-bin-cdh60]# source ~/.bashrc
爲啓動kylin做準備,修改配置文件:
[root@cdh632-worker03 ~]# vi $KYLIN_HOME/conf/kylin.properties
# 設置爲開發環境,便於查看一些新的特性
kylin.env=DEV
### 若多機器部署,可以把第二臺機器設置爲kylin.server.mode=query等
## Kylin server mode, valid value [all, query, job]
#kylin.server.mode=all
kylin.server.cluster-servers=cdh632-worker03:7070
kylin.web.timezone=GMT+8
#kylin.stream.receiver.use-threads-per-query=8
#每個查詢使用四個線程進行掃描
kylin.stream.receiver.use-threads-per-query=4
[root@cdh632-worker03 apache-kylin-3.0.2-bin-cdh60]# $KYLIN_HOME/bin/kylin.sh start
Retrieving hadoop conf dir...
KYLIN_HOME is set to /opt/module/apache-kylin-3.0.2-bin-cdh60
Retrieving hive dependency...
Retrieving hbase dependency...
Hadoop 3.0.0-cdh6.3.2
Retrieving hadoop conf dir...
Retrieving kafka dependency...
Retrieving Spark dependency...
報錯03:
spark not found, set SPARK_HOME, or run bin/download-spark.sh
若成功啓動,則提示如下:
A new Kylin instance is started by root. To stop it, run 'kylin.sh stop'
Check the log at /opt/module/apache-kylin-3.0.2-bin-cdh60/logs/kylin.log
Web UI is at http://cdh632-worker03:7070/kylin
按照其提示,此時查看日誌
[root@cdh632-worker03 ~]# vi /opt/module/apache-kylin-3.0.2-bin-cdh60/logs/kylin.log
發現報錯:
2020-06-25 12:57:54,410 WARN  [localhost-startStop-1] support.XmlWebApplicationContext:551 : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
2020-06-25 12:57:54,416 ERROR [localhost-startStop-1] context.ContextLoader:350 : Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping': Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
        at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
        at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
        at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
        at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
        at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
        at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
        at org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
        at org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
        at org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4699)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5165)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:183)
        at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:743)
        at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:719)
        at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:705)
        at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:970)
        at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1841)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/configuration/ConfigurationException
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
        at java.lang.Class.getDeclaredMethods(Class.java:1975)
        at org.springframework.util.ReflectionUtils.getDeclaredMethods(ReflectionUtils.java:613)
        at org.springframework.util.ReflectionUtils.doWithMethods(ReflectionUtils.java:524)
        at org.springframework.core.MethodIntrospector.selectMethods(MethodIntrospector.java:68)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.detectHandlerMethods(AbstractHandlerMethodMapping.java:230)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.initHandlerMethods(AbstractHandlerMethodMapping.java:214)
        at org.springframework.web.servlet.handler.AbstractHandlerMethodMapping.afterPropertiesSet(AbstractHandlerMethodMapping.java:184)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping.afterPropertiesSet(RequestMappingHandlerMapping.java:127)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
        ... 25 more
Caused by: java.lang.ClassNotFoundException: org.apache.commons.configuration.ConfigurationException
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1358)
        at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1180)
        ... 37 more

經查issues雖報錯不完全相同,但可以察覺是缺jar包,參考網友《kylin啓動打不開web UI》,於是解決方法爲

[root@cdh632-worker03 apache-kylin-3.0.2-bin-cdh60]# find / -name commons-configuration*
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/client/commons-configuration2.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/parquet/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/search/lib/commons-configuration2-2.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/search/lib/search-crunch/commons-configuration2-2.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/solr/server/solr-webapp/webapp/WEB-INF/lib/commons-configuration2-2.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/kite/lib/commons-configuration2-2.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/sentry/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase-solr/lib/commons-configuration2-2.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/embedded-oozie-server/webapp/WEB-INF/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/libtools/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/oozie-sharelib-yarn/lib/git/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/oozie-sharelib-yarn/lib/hcatalog/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/oozie/oozie-sharelib-yarn/lib/pig/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hadoop-hdfs/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/spark/jars/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/hbase/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/lib/impala/lib/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/commons-configuration2-2.1.1.jar
/opt/cloudera/parcels/CDH-6.3.2-1.cdh6.3.2.p0.1605554/jars/commons-configuration2-2.1.jar
/opt/cloudera/cm/cloudera-navigator-audit-server/commons-configuration-1.9.jar
/opt/cloudera/cm/cloudera-navigator-server/jars/commons-configuration-1.9.jar
/opt/cloudera/cm/cloudera-navigator-server/libs/cdh6/commons-configuration2-2.1.1.jar
/opt/cloudera/cm/cloudera-scm-telepub/jars/commons-configuration-1.9.jar
/opt/cloudera/cm/cloudera-scm-telepub/libs/cdh6/commons-configuration2-2.1.1.jar
/opt/cloudera/cm/common_jars/commons-configuration-1.9.cf57559743f64f0b3a504aba449c9649.jar
/opt/cloudera/cm/common_jars/commons-configuration2-2.1.1.0e1d8ed885ef559731c58f49f370fd6e.jar
/opt/cloudera/cm/lib/cdh6/commons-configuration2-2.1.1.jar
/opt/cloudera/cm/lib/commons-configuration-1.9.jar
/opt/module/apache-atlas-sources-2.0.0/addons/storm-bridge/target/dependency/hook/storm/atlas-storm-plugin-impl/commons-configuration2-2.2.jar
/opt/module/apache-atlas-sources-2.0.0/addons/storm-bridge/target/dependency/hook/storm/atlas-storm-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/addons/hbase-bridge/target/dependency/hook/hbase/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/addons/hive-bridge/target/dependency/hook/hive/atlas-hive-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/addons/kafka-bridge/target/dependency/hook/kafka/atlas-kafka-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-kafka-hook/apache-atlas-kafka-hook-2.0.0/hook/kafka/atlas-kafka-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-hive-hook/apache-atlas-hive-hook-2.0.0/hook/hive/atlas-hive-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration2-2.2.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hive/atlas-hive-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-bin/apache-atlas-2.0.0/hook/hbase/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/server/webapp/atlas/WEB-INF/lib/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/server/webapp/atlas/WEB-INF/lib/commons-configuration2-2.1.1.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-server/apache-atlas-2.0.0/hook/hive/atlas-hive-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-storm-hook/apache-atlas-storm-hook-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration2-2.2.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-storm-hook/apache-atlas-storm-hook-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/distro/target/apache-atlas-2.0.0-hbase-hook/apache-atlas-hbase-hook-2.0.0/hook/hbase/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/notification/target/dependency/hook/kafka-topic-setup/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/webapp/target/atlas-webapp-2.0.0/WEB-INF/lib/commons-configuration-1.10.jar
/opt/module/apache-atlas-sources-2.0.0/webapp/target/atlas-webapp-2.0.0/WEB-INF/lib/commons-configuration2-2.1.1.jar
/opt/apache-atlas-2.0.0/hook/hive/atlas-hive-plugin-impl/commons-configuration-1.10.jar
/opt/apache-atlas-2.0.0/hook/hbase/atlas-hbase-plugin-impl/commons-configuration-1.10.jar
/opt/apache-atlas-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration2-2.2.jar
/opt/apache-atlas-2.0.0/hook/storm/atlas-storm-plugin-impl/commons-configuration-1.10.jar
/yarn/nm/filecache/10/3.0.0-cdh6.3.2-mr-framework.tar.gz/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/15/__spark_libs__8545128267598157076.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/19/__spark_libs__4380644035805851635.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/23/__spark_libs__5818005512619487656.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/27/__spark_libs__4877829929501500423.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/31/__spark_libs__3407831335315735358.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/35/__spark_libs__3620082684691436317.zip/commons-configuration2-2.1.1.jar
/yarn/nm/usercache/hdfs/filecache/39/__spark_libs__7765409084065077980.zip/commons-configuration2-2.1.1.jar
考慮到我之前配置的Hive On Spark,那麼先嚐試spark的commons-configuration2-2.1.1.jar卻失敗了(報錯相同)
[root@cdh632-worker03 ~]# $KYLIN_HOME/bin/kylin.sh stop
Stopping Kylin: 34619
Kylin with pid 34619 has been stopped.
[root@cdh632-worker03 ~]# cp /yarn/nm/filecache/10/3.0.0-cdh6.3.2-mr-framework.tar.gz/commons-configuration2-2.1.1.jar $KYLIN_HOME/lib/
於是老老實實使用commons-configuration-1.9.jar吧
[root@cdh632-worker03 ~]# $KYLIN_HOME/bin/kylin.sh stop
[root@cdh632-worker03 ~]# cp /opt/cloudera/cm/lib/commons-configuration-1.9.jar $KYLIN_HOME/lib/
[root@cdh632-worker03 ~]# $KYLIN_HOME/bin/kylin.sh start

按照啓動控制檯日誌上的提示,打開首頁:http://cdh632-worker03:7070/kylin,輸入賬號/密碼:ADMIN/KYLIN

注意:首頁url必須帶kylin否則還會報404錯誤,賬號和密碼均爲大寫。

參考:

CDH 6.3.1 部署 Kylin 3.0.1

kylin在啓動後web ui無法打開

問題一:由於本人Hive On Spark,在創建cube時“Advanced Setting”的“Cube Engine”處Engine Type應選Spark而非默認的mapreduce,但選擇spark後,“actions”選擇build後狀態顯示ERROR,此時發現運行到第七步“Build Cube with Spark”報錯如下:

20/06/26 11:56:09 ERROR spark.SparkContext: Error initializing SparkContext.
java.lang.IllegalArgumentException: Required executor memory (4096), overhead (1024 MB), and PySpark memory (0 MB) is above the max threshold (2048 MB) of this cluster! Please check the values of 'yarn.scheduler.maximum-allocation-mb' and/or 'yarn.nodemanager.resource.memory-mb'.

解決方案:按照日誌中的提示,在yarn頁面的“配置”處搜索此兩個參數並將其設置爲5G(兩個都要修改且至少爲5G)。

問題二:kylin官網文檔Use Dashboard顯示此功能在2.3.0版本之後就支持了,但我按官網配置後chart顯示報錯,待解決。。。

[root@cdh632-worker03 ~]# $KYLIN_HOME/bin/kylin.sh stop
[root@cdh632-worker03 ~]# vi /opt/module/apache-kylin-3.0.2-bin-cdh60/conf/kylin.properties
隨便找一個web設置處添加:kylin.web.dashboard-enabled=true
[root@cdh632-worker03 ~]# $KYLIN_HOME/bin/kylin.sh start

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章