doris 部署總結

最近調研doris 吧一些安裝和部署的細節做一下記錄吧 以後可以用：
首先是安裝 建議用docker 11的版本在Linus安裝一直都有問題 在docker就可以成功安裝：
具體步驟：
1. pull doris dev docker image

sudo docker pull apachedoris/doris-dev:build-env-1.1
 

2. dowload doris-0.11.20 source local

mkdir -p /home/lenmom/wget https://codeload.github.com/baidu-doris/incubator-doris/tar.gz/DORIS-0.11.20-release
tar -xzvf incubator-doris-DORIS-0.11.20-release.tar.gz
mv incubator-doris-DORIS-0.11.20-release  doris-0.11.20-release

3.build

sudo docker  run -it  -v /home/lenmom/doris-0.11.20-release:/root/doris-0.11.20   --name doris_dev  apachedoris/doris-dev:build-env-1.1  //bin/bash

export PATH=$PATH:$JAVA_HOME/bin:$MAVEN_HOME/bin && cd /root/doris-0.11.20 && sh build.sh

大概一個小時多一點差不多了 但是在自己主機上好久都沒有build成功
這裏要注意有一個地方要改一下：
項目fe目錄下的pom文件 有一個

           <pluginRepositories>
                <!-- for cup-maven-plugin -->
                <pluginRepository>
                    <id>spring-plugins</id>
                    <url>**https** : //repo.spring.io/plugins-release/</url>
                </pluginRepository>
            </pluginRepositories>

http要改爲加s的不然就會報錯

到這裏最基礎的算是安裝完成了
但是如果要想重hdfs上load表 還需要buildbroker
那就來到docker裏 執行以下sh build.sh
這裏在docker裏好像沒有什麼錯誤 但是在Linus上還是會有錯誤 這個就要具體情況具體分析了。

然後是部署
首先我們要部署fe節點把你第一次build完成的最外面的那個output文件夾下fe裏的文件和文件夾cp到fe裏面去 改fe文件夾裏conf文件夾下的配置文件

# store metadata, create it if it is not exist.
meta_dir = ${DORIS_HOME}/palo-meta #這個目錄要自己創建

http_port = 8030
rpc_port = 9020
query_port = 9030 --這個是你mysql接入需要的端口
edit_log_port = 9010# 注意這些端口要確定現在沒有被佔用不然會啓動失敗

# Choose one if there are more than one ip except loopback address. 
# Note that there should at most one ip match this list.
# If no ip match this rule, will choose one randomly.
# use CIDR format, e.g. 10.10.10.0/24
# Default value is empty.
priority_networks = 172.30.22.80/24#這個fe節點的ip

# Advanced configurations 
# log_roll_size_mb = 1024
# sys_log_dir = ${DORIS_HOME}/log #可以在這裏查看日誌 這個目錄自己生成
# sys_log_roll_num = 10
# sys_log_verbose_modules = 
# audit_log_dir = ${DORIS_HOME}/log
# audit_log_modules = slow_query, query
# audit_log_roll_num = 10
# meta_delay_toleration_second = 10
# qe_max_connection = 1024
# max_conn_per_user = 100
# qe_query_timeout_second = 300
# qe_slow_log_ms = 5000

然後就可以啓動了
mysql -p 9030 -h xxxxxx -u root
這裏是不用密碼的密碼可以進入之後自己設置
具體設置等以後補充

然後是be節點把你第一次build完成的最外面的那個output文件夾下be裏的文件和文件夾cp到be裏面去 改be文件夾裏conf文件夾下的配置文件
然後也是配置節點 在裏面加上這一個注意這個要自己創建目錄在自己的be裏main創建

storage_root_path = /home/tujia/doris-0.11.20-release/palo

然後是broker
hdfs-site.xml 把你集羣對應的內容寫進去
apache_hdfs_broker.conf這個裏面的端口配置好方便一會用

···

添加borker節點

ALTER SYSTEM ADD BROKER broker1 "173.29.40.42:8000";

添加be節點

ALTER SYSTEM ADD BACKEND "172.29.40.42:9050";

查看be broker

···
show proc “/backends”
SHOW PROC “/brokers”;
···

mysql -h 172.29.40.42 -P9030 -uroot
sudo netstat -tunlp
docker
docker run -it containerid /bin/bash
docker exec -it containerid /bin/bash
docker cp 文件 containerid ：docker文件

CREATE TABLE IF NOT EXISTS example_db.expamle_tbl
(
user_id LARGEINT NOT NULL COMMENT “用戶id”,
date DATE NOT NULL COMMENT “數據灌入日期時間”,
timestamp DATETIME NOT NULL COMMENT “數據灌入的時間戳”,
city VARCHAR(20) COMMENT “用戶所在城市”,
age SMALLINT COMMENT “用戶年齡”,
sex TINYINT COMMENT “用戶性別”,
last_visit_date DATETIME REPLACE DEFAULT “1970-01-01 00:00:00” COMMENT “用戶最後一次訪問時間”,
cost BIGINT SUM DEFAULT “0” COMMENT “用戶總消費”,
max_dwell_time INT MAX DEFAULT “0” COMMENT “用戶最大停留時間”,
min_dwell_time INT MIN DEFAULT “99999” COMMENT “用戶最小停留時間”
)
ENGINE=olap
AGGREGATE KEY(user_id, date, timestamp, city, age, sex)
PARTITION BY RANGE(date)
(
PARTITION p201701 VALUES LESS THAN (“2017-02-01”),
PARTITION p201702 VALUES LESS THAN (“2017-03-01”),
PARTITION p201703 VALUES LESS THAN (“2017-04-01”)
)
DISTRIBUTED BY HASH(user_id) BUCKETS 16
PROPERTIES
(
“replication_num” = “3”,
“storage_medium” = “SSD”,
“storage_cooldown_time” = “2018-01-01 12:00:00”
);

doris 配置
storage_root_path = /home/hadoopcdh/incubator-doris-DORIS-0.11.33.1-release/be/palo
priority_networks = 172.29.40.42/24;192.168.0.0/16
ALTER SYSTEM ADD BACKEND “172.29.40.42:9050”;
show proc “/backends”
SHOW PROC “/brokers”;
ALTER SYSTEM ADD BROKER broker1 “173.29.40.42:8000”;
ALTER SYSTEM drop BROKER broker1 “173.29.40.42:8000”;

scp -r local_folder remote_username@remote_ip:remote_folder
或者
scp -r local_folder remote_ip:remote_folder

···