presto on yarn生產實踐

presto on yarn方案缺點

hadoop集羣的jdk版本過低無法滿足prestoserver需求
配置文件不易維護，特別是如果涉及多個hadoop集羣，這點在大公司很常見
hadoop集羣單獨團隊維護，自己沒有權限創建目錄之類的，這個也很常見
指定coordinator節點需要使用yarn label，否則每次重啓都得找coordinator節點

針對上述問題，解決方案如下：hadoop集羣jdk版本低的問題，可以嘗試將高版本jdk打包到prestoserver中去，啓動的時候用自帶的jdk；配置文件也可以和prestoserver一起打包，不使用appConfig.conf這種方式配置；presto中的數據目錄打算在程序slider啓動時，在tmp目錄創建，不再手動創建；presto on yarn這種方式，我們只部署worker節點；coordinator單獨找一臺物理機部署
版本如下

hadoop version	presto version	slider version	jdk version
cdh5.15.2	0.223	slider-0.80.0-incubating	jdk1.8.0_181

下載編譯presto-yarn

git clone https://github.com/prestodb/presto-yarn.git

修改presto.verison和slider.version

vi pom.xml

<presto.version>0.223</presto.version>
<slider.version>0.80.0-incubating</slider.version>

修改configure.py

#!/usr/bin/env python

from resource_management import *
import ast, os, shutil

def set_configuration(component=None):

    import params

    if (os.path.exists(format("{data_dir}"))):
        shutil.rmtree(format("{data_dir}"))

    if (os.path.lexists(format("{conf_dir}"))):
        os.remove(format("{conf_dir}"))

    _directory(params.pid_dir, params)
    _directory(params.log_dir, params)
 

    if params.addon_plugins:
        plugins_dict = ast.literal_eval(params.addon_plugins)
        for key, value in plugins_dict.iteritems():
            plugin_dir = os.path.join(params.presto_plugin_dir, key)
            if not os.path.exists(plugin_dir):
                os.makedirs(plugin_dir)
            for jar in value:
                shutil.copy2(os.path.join(params.source_plugin_dir, jar), plugin_dir)
            
def _directory(path, params):
    Directory(path,
              owner=params.presto_user,
              group=params.user_group,
              recursive=True
              )

修改presto_server.py

#!/usr/bin/env python

from resource_management import *
from configure import set_configuration
import os

class PrestoServer(Script):
    def __init__(self, component):
        self.component = component

    def install(self, env):
        self.install_packages(env)
        pass

    def configure(self):
        set_configuration(self.component)

    def start(self, env):
        import params

        env.set_params(params)

        self.configure()
        #修改此處
        # os.symlink(format('{conf_dir}'), os.path.join(format('{presto_root}'), 'etc'))
        os.symlink(os.path.join(format('{presto_root}'), 'etc'),format('{conf_dir}'))
        process_cmd = format("PATH={presto_root}/jdk1.8.0_181/bin:$PATH {presto_root}/bin/launcher run --node-config {conf_dir}/node.properties --jvm-config {conf_dir}/jvm.config --config {conf_dir}/config.properties >> {log_file} 2>&1")

        Execute(process_cmd,
                logoutput=True,
                wait_for_finish=False,
                pid_file=params.pid_file,
                poll_after=3
                )

    def stop(self, env):
        pass

    def status(self, env):
        import params

        env.set_params(params)
        check_process_status(params.pid_file)


if __name__ == "__main__":
    self.fail_with_error('Component name missing')

打包編譯

mvn clean package -DskipTests

編譯完後的包在presto-yarn-package/target目錄下

/Users/xiao/IdeaProjects/presto-yarn/presto-yarn-package/target/presto-yarn-package-1.6-SNAPSHOT-0.223.zip

打完包後presto-yarn-package-1.6-SNAPSHOT-0.223.zip中的prestoserver用的是官網的安裝包，不包含任何配置文件，所以需要解壓後替換成包含配置文件的package，同時將jdk也放到prestoserver中去，然後重新打包，這個過程就不演示了,我的prestoserver package結構如下

[yarn@hadoop6 presto-server-0.223]$ ll
total 220
drwxr-xr-x  3 yarn yarn   4096 Jul 26 04:17 bin
drwxr-xr-x  3 yarn yarn   4096 Aug  7 11:45 etc
drwxr-xr-x  7 yarn yarn   4096 Aug  7 14:17 jdk1.8.0_181
drwxrwxr-x  2 yarn yarn  12288 Aug  7 09:41 lib
-rw-r--r--  1 yarn yarn 191539 Jul 26 03:58 NOTICE
drwxrwxr-x 30 yarn yarn   4096 Aug  7 09:41 plugin
-rw-r--r--  1 yarn yarn    126 Jul 26 03:58 README.txt

下載配置slider

地址https://archive.apache.org/dist/incubator/slider/,這裏我們下載的版本是0.80.0-incubating,一定要選擇和自己hadoop版本一致的版本，這裏下載的是已經編譯好的包

wget https://archive.apache.org/dist/incubator/slider/0.80.0-incubating/slider-assembly-0.80.0-incubating-all.zip

解壓

unzip slider-assembly-0.80.0-incubating-all.zip

配置slider

vi slider-env.sh

export JAVA_HOME=/usr/local/java/jdk1.8.0_181
export HADOOP_CONF_DIR=/etc/hadoop/conf

vi slider-client.xml

<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<configuration>
    <property>
      <name>slider.client.resource.origin</name>
      <value>conf/slider-client.xml</value>
      <description>This is just for diagnostics</description>
    </property>

    <property>
      <name>yarn.log-aggregation-enable</name>
      <value>true</value>
    </property>

    <property>
        <name>slider.security.protocol.acl</name>
      <value>*</value>
      <description>When security is enabled, set appropriate acl. Default value means allow everyone.</description>
    </property>

    <property>
      <name>yarn.application.classpath</name>
      <value>$HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*</value>
    </property>

    <property>
      <name>yarn.resourcemanager.address</name>
      <value>Hadoop:8032</value>
    </property>

    <property>
      <name>yarn.resourcemanager.scheduler.address</name>
      <value>hadoop:8030</value>
    </property>

    <property>
      <name>fs.defaultFS</name>
      <value>hdfs://hadoop:8020</value>
    </property>

    <property>
      <name>hadoop.registry.zk.quorum</name>
      <value>hadoop1:2181,hadoop2:2181,hadoop3:2181,hadoop4:2181,hadoop5:2181</value>
    </property>

</configuration>

presto 包安裝

[yarn@hadoop6 ~]$ slider-0.80.0-incubating/bin/slider package --install --name presto --package presto-yarn-package-1.6-SNAPSHOT-0.223.zip
2019-08-06 15:15:21,690 [main] INFO  client.SliderClient - Installing package file:/var/lib/hadoop-yarn/presto-yarn-package-1.6-SNAPSHOT-0.223.zip to hdfs://hadoop:8020/user/yarn/.slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.223.zip (overwrite set to false)
2019-08-06 15:15:30,805 [main] INFO  client.SliderClient - Set application.def in your app config JSON to .slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.223.zip
2019-08-06 15:15:30,807 [main] INFO  util.ExitUtil - Exiting with status 0

配置slider
vi slider-0.80.0-incubating/conf/appConfig.json

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "site.global.app_user": "yarn",
    "site.global.user_group": "hadoop",
    "site.global.data_dir": "/tmp/presto223/data",
    "site.global.config_dir": "/tmp/presto223/etc",
    "site.global.app_name": "presto-server-0.223",
    "site.global.app_pkg_plugin": "${AGENT_WORK_ROOT}/app/definition/package/plugins/",
    "site.global.singlenode": "false",
    "site.global.coordinator_host": "coodinatorhost 節點ip",
    "site.global.presto_server_port": "8092",
    "site.global.catalog":"",
    "application.def": ".slider/package/presto/presto-yarn-package-1.6-SNAPSHOT-0.223.zip"
  },
  "components": {
    "slider-appmaster": {
      "jvm.heapsize": "128M"
    }
  }
}

vi slider-0.80.0-incubating/conf/resources.json

{
  "schema": "http://example.org/specification/v2.0.0",
  "metadata": {
  },
  "global": {
    "yarn.vcores": "1"
  },
  "components": {
    "slider-appmaster": {
    },
    "WORKER": {
      "yarn.role.priority": "2",
      "yarn.component.instances": "7",
      "yarn.component.placement.policy": "2",
      "yarn.memory": "12000"
    }
  }
}

部署coordinator 節點
找臺物理機單獨部署coordinator，這部分略過

啓動worker

###第一次執行忽略第一條命令，多次執行的時候需要刪除hdfs上的緩存安裝包
hdfs dfs -rmr -skipTrash hdfs://hadoop1:8020/user/yarn/.slider/cluster/prestoserver

./bin/slider create prestoserver --template conf/appConfig.json --resources conf/resources.json

查看slider日誌，可以看到使用的是presto自帶的jdk

2019-08-07 14:36:13,686 - Execute['PATH=/data2/yarn/nm/usercache/yarn/appcache/application_1565086656619_0056/container_1565086656619_0056_01_000008/app/install/presto-server-0.223/jdk1.8.0_181/bin:$PATH /data2/yarn/nm/usercache/yarn/appcache/application_1565086656619_0056/container_1565086656619_0056_01_000008/app/install/presto-server-0.223/bin/launcher run --node-config /tmp/presto223/etc/node.properties --jvm-config /tmp/presto223/etc/jvm.config --config /tmp/presto223/etc/config.properties >> /tmp/presto223/data/var/log/server.log 2>&1'] {'pid_file': '/tmp/presto223/data/var/run/slider_launcher.pid', 'wait_for_finish': False, 'logoutput': True, 'poll_after': 3}

控制檯

問題

1:python環境問題
要確保python環境統一，這裏使用系統自帶的
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type “help”, “copyright”, “credits” or “license” for more information.
如果有些節點一直部署不成功，檢查yarn container日誌，很有可能是python環境問題，例如錯誤：CERTIFICATE_VERIFY_FAILED

2.啓動的server實例少於配置所指定
原因有很多，在我配置過程中總結了下大概有如下幾點：

端口被佔用，主要是由於一個nodemanger啓動多個presto的containner，通常我會讓container的內存大於nodemanager最大內存一半以上，這樣每個nodemanager就只能啓動一個prestoserver
yarn沒有足夠的資源分配給prestoserver，主要體現在nodemanager的可用內存不夠啓動prestoserver，也有可能是nodemanager cpu 全被佔用，這些都是可以在控制檯查看：http://hadoop2:8088/cluster/nodes

3.日誌查看

通常情況下slider日誌提供的信息非常有限，儘量查看yarn的日誌，yarn logs -applicationId appid

presto on yarn生產實踐

下載編譯presto-yarn

下載配置slider

配置slider

PDManer [元數建模]-v4.9.0 發佈：一款簡單好用的數據庫建模平臺

使用neovim打造go ide(支持代碼跳轉, 代碼補全, 實時語法檢查)

sql求連續值問題

cs01 CSS Syntax

挑戰程序設計競賽 2.3章習題 poj 3046 Ant Counting

[MASM拾遺]Offset僞指令

h30 HTML Layout Elements

瞭解顯卡

一款基於C#開發的通訊調試工具（支持Modbus RTU、MQTT調試）

Linux/Golang/glibC系統調用

阿里雲上安裝FreeIPA（ipa-server && ipa-client）

實時數倉設計之mysql binlog

從checkpoint恢復flink job

Neo.TransientError.Transaction.DeadlockDetected

kudu master uuid異常修復

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結