Presto on yarn解決方案

原創 panguoyuan 2018-09-03 20:10

Deploying Presto on a YARN-Based Cluster

presto不像spark那樣默認就支持yarn，spark與yarn兼容性很好， 只需要簡單的配置下啓動腳本和集羣環境就可以在Yarn上運行spark任務。presto則不然它需要藉助於slider。通過slider實現prestoon yarn。Yarn是一個通用資源管理系統，可爲上層應用提供統一的資源管理和調度，它的引入爲集羣在利用率、資源統一管理和數據共享等方面帶來了巨大好處。所以此方案就是把presto的應用提交到yarn上。prestoon yarn不可以直接使用官方提供的二進制包安裝，需要重新編譯presto，編譯生成presto-yarn-package-1.6-SNAPSHOT-0.184.zip，通過slider來使用這個包。網上資料確實很少，基本上找不到presto on yarn的資料，只能依靠並不是很詳細的官方文檔，遇到錯之後再排查。如果hadoop是cdh版編譯slider時需要指定cdh版本默認是apache hadoop。譯安裝部署過程中遇到很多坑，現在把編譯、安裝、調試部署的過程記錄下來僅供參考。

 

一、環境基本要求

•  Linux

•  Java 8, 64-bit

•  Presto-0.184

•  Zookeeper

•  Hadoop

•  Yarn

二、Presto基礎配置及架構

•  Presto架構圖


•  presto執行過程示意圖


•  連接器

•   Presto支持從以下版本的Hadoop中讀取Hive數據：支持以下文件類型：Text, SequenceFile, RCFile, ORC

1                   Apache Hadoop 1.x （hive-hadoop1）

2                   Apache Hadoop 2.x  （hive-hadoop2）

3                   Cloudera CDH4      （hive-cdh4）

4                   Cloudera CDH5      （hive-cdh5）

•   此外，需要有遠程的Hive元數據。不支持本地或嵌入模式。 Presto不使用MapReduce，只需要HDFS

三、安裝配置步驟

1、編譯presto-on-yarn

（1）下載地址：https://github.com/prestodb/presto-yarn/

（2）編譯：mvn clean package -Dpresto.version=0.184


從上面可以看出已經編譯完了，在編譯完的目錄下可以找到presto-yarn-package-1.6-SNAPSHOT-0.184.zip

查看presto-yarn-package-1.6-SNAPSHOT-0.184.zip的目錄結構


2、編譯slider

下載地址：https://archive.apache.org/dist/incubator/slider/

編譯slider，指定CDH版本，升級JDK

https://slider.incubator.apache.org/developing/building.html

(1)下載源碼：apache-slider-0.91.0-incubating-source-release.tar.gz

(2)修改maven相關的配置

採用CDH版本hadoop(2.6.0-cdh5.7.0)，hbase(1.2.0-cdh5.7.0)，修改該目錄下pom文件

<hadoop.version>2.6.0-cdh5.7.0</hadoop.version>
<hbase.version>1.2.0-cdh5.7.0</hbase.version>
<accumulo.version>1.7.0</accumulo.version>

註釋掉slider-core和slider-funtest中對hadoop-minicluster包依賴

<!--
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-minicluster</artifactId>
<scope>test</scope>
</dependency>
-->

（3）編譯：mvn clean package -Dmaven.test.skip=true -DskipTests


在編譯後的目錄下找到slider-0.91.0-incubating-all.tar.gz

（4）slider配置，參照官方詳細文檔

https://prestodb.io/presto-yarn/installation-yarn-manual.html#deploying-presto-on-a-yarn-based-cluster，解壓後，在conf目錄下找到下面兩個配置文件

slider-client.xml

<configuration>
    <property>
      <name>slider.client.resource.origin</name>
      <value>conf/slider-client.xml</value>
      <description>This is just for diagnostics</description>
    </property>
    <property>
      <name>slider.security.protocol.acl</name>
      <value>*</value>
    </property>
    <property>
      <name>slider.yarn.queue</name>
      <value>root.presto</value>
      <description>the name of the YARN queue to use.</description>
    </property>
    <property>
      <name>slider.yarn.queue.priority</name>
      <value>1</value>
      <description>the priority of the application.</description>
    </property>
    <property>
      <name>slider.am.login.keytab.required</name>
      <value>false</value>
      <description>Declare that a keytab must be provided.</description>
    </property>
   <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
    </property>
    <property>
      <name>yarn.application.classpath</name>
    <value>$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/share/hadoop/common/*,$HADOOP_COMMON_HOME/share/
hadoop/commo
n/lib/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/*,$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*,$HADOOP_YA
RN_HOME/shar
e/hadoop/yarn/*,$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*</value>
    </property>
    <property>
      <name>yarn.resourcemanager.address</name>
      <value>master1:8032</value>
    </property>
    <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>master2:8030</value>
    </property>
    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://common/user/hadoop/.slider/</value>
    </property>
     <property>
       <name>slider.zookeeper.quorum</name>
       <value>node1:2181,node2:2181,node3:2181</value>
    </property>
</configuration>

slider-env.sh配置

export JAVA_HOME=${JAVA_HOME}
export HADOOP_CONF_DIR=${HADOOP_CONF_DIR}
export SLIDER_JVM_OPTS="-server -Xmx40g -Xms4g -Xmn8g"

解壓presto-yarn-package-1.6-SNAPSHOT-0.184.zip，獲取appConfig-default.json，resources-default.json，修改裏面的配置項設置分配presto的資源, 相關配置參照如下(官方配置說明文檔https://prestodb.io/presto-yarn/installation-yarn-configuration-options.html)，appConfig.json配置

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "java_home": "/usr/local/java",
    "site.global.app_user": "hadoop",
    "site.global.user_group": "hadoop",
    "site.global.app_name": "presto-server-0.184",
    "site.global.data_dir": "/data01/presto/data",
    "site.global.config_dir": "/data01/presto/etc",
    "zookeeper.quorum" : "node1:2181,node2:2181,node3:2181",
    "application.def": "hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-S
NAPSHOT-0.184.zip",
    "site.global.singlenode": "false",
    "site.global.coordinator_host": "${COORDINATOR_HOST}",
    "site.global.app_pkg_plugin": "${AGENT_WORK_ROOT}/app/definition/package/plugins/",
    "site.global.presto_query_max_memory": "512GB",
    "site.global.presto_query_max_memory_per_node":"20GB",
    "site.global.presto_server_port": "18088",
    "site.global.catalog": "{'hive': ['connector.name=hive-hadoop2','hive.config.resources=/usr/lo
cal/hadoop/etc/hadoop/core-site.xml,/usr/local/hadoop/etc/hadoop/hdfs-site.xml','hive.metastore.ur
i=thrift://10.134.81.70:9083'], 'tpch': ['connector.name=tpch']}",
    "site.global.jvm_args": "['-server', '-Xmx40960M', '-XX:+UseG1GC', '-XX:G1HeapRegionSize=320M'
, '-XX:+UseGCOverheadLimit', '-XX:+ExplicitGCInvokesConcurrent', '-XX:+HeapDumpOnOutOfMemoryError'
, '-XX:OnOutOfMemoryError=kill -9 %p']",
    "site.global.log_properties": "['com.facebook.presto.hive=WARN','com.facebook.presto.server=IN
FO']",
    "site.global.additional_config_properties":"['task.max-worker-threads=50', 'distributed-joins-
enabled=true']"
  },
  "components": {
    "slider-appmaster": {
      "jvm.heapsize": "1024M"
    },
    "MYAPP_COMPONENT": {
    }
  }
}
 
{
  "schema" : "http://example.org/specification/v2.0.0",
  "metadata" : {
  },
  "global" : {
     "yarn.vcores": "1"
  },
  "components": {
    "slider-appmaster": {
    },
   "COORDINATOR": {
      "yarn.role.priority": "1",
      "yarn.component.instances": "1",
      "yarn.component.placement.policy": "1",
      "yarn.memory": "20000"
    },
    "WORKER": {
      "yarn.role.priority": "2",
      "yarn.component.instances": "20",
      "yarn.component.placement.policy": "1",
      "yarn.memory": "20000"
    }
  }
}

3、使用slider安裝presto on yarn

在hdfs創建slider的相關目錄及授予訪問權限，確保當前用戶有訪問權限，在每個nodemanager創建presto的本地目錄

[hadoop@master1 slider]$ slider package --install --name presto --package presto-yarn-package-1.6-SNAPSHOT-0.184.zip  
2017-09-25 14:23:00,327 [main] INFO  client.SliderClient - Installing package file:/usr/local/apache/slider-0.91.0-incubating/presto-yarn-package-1.6-SNAPSHOT-0.184.zip to hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip (overwrite set to false)
2017-09-25 14:23:03,114 [main] INFO  tools.SliderUtils - Reading metainfo.xml of size 2425
2017-09-25 14:23:03,115 [main] INFO  client.SliderClient - Found XML metainfo file in package
2017-09-25 14:23:03,135 [main] INFO  client.SliderClient - Creating summary metainfo file
2017-09-25 14:23:03,153 [main] INFO  client.SliderClient - Set application.def in your app config JSON to .slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip
2017-09-25 14:23:03,154 [main] INFO  util.ExitUtil - Exiting with status 0

4、啓動Presto OnYarn 

[hadoop@master1 slider]$ slider create presto-query --template appConfig.json --resources resources.json        
2017-09-25 15:06:35,776 [main] INFO  agent.AgentClientProvider - Validating app definition hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip
2017-09-25 15:06:35,778 [main] INFO  agent.AgentUtils - Reading metainfo at hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip
2017-09-25 15:06:35,945 [main] INFO  agent.AgentUtils - Got metainfo from summary file
2017-09-25 15:06:35,985 [main] INFO  client.SliderClient - No credentials requested
2017-09-25 15:06:36,087 [main] INFO  agent.AgentUtils - Reading metainfo at hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip
2017-09-25 15:06:36,094 [main] INFO  agent.AgentUtils - Got metainfo from summary file
2017-09-25 15:06:36,127 [main] INFO  launch.AbstractLauncher - Setting yarn.resourcemanager.am.retry-count-window-ms to 300000
2017-09-25 15:06:36,127 [main] INFO  launch.AbstractLauncher - Log include patterns: 
2017-09-25 15:06:36,127 [main] INFO  launch.AbstractLauncher - Log exclude patterns: 
2017-09-25 15:06:36,461 [main] INFO  slideram.SliderAMClientProvider - Loading all dependencies for AM.
2017-09-25 15:06:36,462 [main] INFO  tools.SliderUtils - Loading all dependencies from /usr/local/apache/slider-0.91.0-incubating/lib
2017-09-25 15:06:40,709 [main] INFO  agent.AgentClientProvider - Automatically uploading the agent tarball at hdfs://common/user/hadoop/.slider/cluster/presto-query/tmp/application_1504914229457_101524/agent
2017-09-25 15:06:40,791 [main] INFO  agent.AgentClientProvider - Validating app definition hdfs://common/user/hadoop/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.184.zip
2017-09-25 15:06:40,796 [main] INFO  tools.SliderUtils - For faster submission of apps, upload dependencies using cmd dependency --upload
2017-09-25 15:06:40,804 [main] INFO  client.SliderClient - Submitting application application_1504914229457_101524
2017-09-25 15:06:40,808 [main] INFO  launch.AppMasterLauncher - Submitting application to Resource Manager
2017-09-25 15:06:41,036 [main] INFO  impl.YarnClientImpl - Submitted application application_1504914229457_101524
2017-09-25 15:06:41,039 [main] INFO  util.ExitUtil - Exiting with status 0

在Yarn的後臺可以看到剛纔啓動的application

在zk裏也可以看到對應的presto應用


5、簡單的使用方法

（1）從yarn界面找到coordinator_address地址

（2）在命令行使用presto

 

問題總結

問題1：

NetUtil.py:62 - SSLError: Failed to connect. Please check openssl library versions

解決：需要升級openssl(openssl>= 1.0.1e-16)

（1）         查看當前openssl版本：rpm -qa | grep openssl，如果版本低的話需要升級yum -y upgrade openssl

（2）         升級openssl，重啓nodemanager

問題2：  presto對jdk版本有要求，需要升級jdk版本

問題3：編譯slider指定cdh版本的時候可能maven依賴或中央倉庫連不上等錯誤

解決：觀察maven對應的錯誤信息解決即可，因爲不同的環境編譯報是不一樣的

問題4：resouceConf.json配置的jvm參數太小，當查詢大點的SQL時會報錯

問題5：slider默認的參數可能與現在集羣的環境不匹配，導致一些參數不可用導致報錯，如：yarn.label.expression，只有當使用CapacityScheduler模式時才能使用這屬性指定coordinator和worker，否則會報錯；目前YARN使用的調度模式是Fair Scheduler，所以不支持。

問題6：appConfig.json 文件裏的site.global.catalog配置項，默認很多參數都沒有，需要對presto很熟悉遇到錯誤才馬上定位問題所在，比如：connector.name =hive-hadoop2，hive.config.resources= =/usr/lo

cal/hadoop/etc/hadoop/core-site.xml,/usr/local/hadoop/etc/hadoop/hdfs-site.xml'。如果connector.name 值是hive-cdh5會報錯；默認沒有hive.config.resources配置項，如果不配這項識別不到HDFS環境。

問題7：當啓動多個presto application時，需要注意相關配置項，要不會與之前啓過的application衝去導致影響別的服務，因爲每啓動一個application都會先刪除對應的data和etc目錄，然後再重新生成配置和數據目錄

問題8：site.global.jvm_args參數設置的值要大於site.global.presto_query_max_memory_per_node的值，否則報OOM

問題9：內存單位寫成g報錯，需要寫成GB

問題10：....


發表評論
所有評論
還沒有人評論，想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章

Kafka Manager安裝筆記

1、下載已經編譯好的kafka-manager-1.0-SNAPSHOT.zip   百度網盤：http://pan.baidu.com/s/1i349zet 密碼：9vfk 2、解壓到指定的安裝目錄：unzip kafka-manage

panguoyuan 2020-06-22 00:16:51

Storm啓動異常[ERROR] Error when processing event，Supervisor啓不來

panguoyuan 2020-02-27 09:49:26

Kafka監控工具KafkaOffsetMonitor

panguoyuan 2020-02-27 09:49:26

Flume使用筆記

panguoyuan 2020-02-27 09:49:26

Spark集羣基於Zookeeper的HA搭建部署筆記

panguoyuan 2020-02-27 09:49:26

hadoop配置好之後啓服務，jps能看到datanode進程，可是後臺的datanode日誌有如下錯誤，且50070端口上也是沒有活的節點

panguoyuan 2020-02-27 09:49:26

JavaAPI訪問Hadoop2.2HA的配置下訪問Hbase0.96.2

panguoyuan 2020-02-27 09:49:26

自定義FlumeKafkaSink

panguoyuan 2020-02-27 09:49:26

storm提交topology時supervisor無法創建worker

panguoyuan 2020-02-27 09:49:26

自定義HbaseSink輸出採集日誌到Hbase

panguoyuan 2020-02-27 09:49:26

Spark-2.1.0-hadooop-2.6.0-cdh5.7.0源碼編譯

panguoyuan 2018-09-03 20:10:30

eclipse-spark開發環境

panguoyuan 2018-09-03 20:10:30

nginx+tomcat負載均衡實現session共享

panguoyuan 2018-09-03 20:10:27