Druid安裝部署
一、環境需求
Java 8 （8u92 +）
Linux, Mac OS X, or other Unix-like OS（不支持Windows系統）
Zookeeper (3.4 +)
Druid下載：
官網：https://druid.apache.org，進入官網後點擊“Download”,進入下載界面，當前最新版本0.16，這裏我們選擇編譯版本下載；
如果想下載歷史版本，可以下拉，點擊“Apache release archives “進入歷史版本下載界面
二、單機版安裝部署
1.將安裝包上傳到服務器，並解壓安裝包
tar -xzf apache-druid-0.16.0-incubating-bin.tar.gz -C /data
2.下載zookeeper安裝包上傳到服務器，並解壓安裝包到druid的根目錄，並重名爲zk
tar -xzf zookeeper-3.4.6.tar.gz -C /data/apache-druid-0.16.0-incubating
mv /data/apache-druid-0.16.0-incubating/zookeeper-3.4.6 /data /apache-druid-0.16.0-incubating/zk
3.進入druid的安裝目錄，執行單機啓動腳本
cd /data/apache-druid-0.16.0-incubating
./bin/start-micro-quickstart
4.訪問http://localhost:8888 查看druid管理界面
三、集羣版安裝部署
部署規劃
|
角色 |
機器 |
集羣角色(進程) |
|
主節點 |
hadoop1（192.168.252.111） |
Coordinator,Overlord |
|
數據節點 |
hadoop2（192.168.252.112） |
Historical, MiddleManager |
|
查詢節點 |
hadoop3（192.168.252.113） |
Broker ,router |
外部依賴
|
組件 |
機器 |
描述 |
|
zookeeper |
hadoop1、hadoop2、hadoop3 |
提供分佈式協調服務 |
|
hadoop |
hadoop1、hadoop2、hadoop3 |
提供數據文件存儲庫 |
|
mysql |
hadoop1 |
提供元數據存儲庫 |
注：外部依賴組件安裝請查閱相關資料，這裏默認外部依賴已安裝成功並運行
1.將安裝包上傳到hadoop1服務器，並解壓安裝包，並進入該目錄
tar -xzf apache-druid-0.16.0-incubating-bin.tar.gz -C /data
cd /data/apache-druid-0.16.0-incubating
2.修改核心配置文件，配置文件如下
vim conf/druid/cluster/_common/common.runtime.properties
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
#
# Extensions specified in the load list will be loaded by Druid
# We are using local fs for deep storage - not recommended for production - use S3, HDFS, or NFS instead
# We are using local derby for the metadata store - not recommended for production - use MySQL or Postgres instead
# If you specify `druid.extensions.loadList=[]`, Druid won't load any extension from file system.
# If you don't specify `druid.extensions.loadList`, Druid will load all the extensions under root extension directory.
# More info: https://druid.apache.org/docs/latest/operations/including-extensions.html
druid.extensions.loadList=["mysql-metadata-storage","druid-hdfs-storage","druid-kafka-indexing-service"]
# If you have a different version of Hadoop, place your Hadoop client jar files in your hadoop-dependencies directory
# and uncomment the line below to point to your directory.
#druid.extensions.hadoopDependenciesDir=/my/dir/hadoop-dependencies
#
# Hostname
#
druid.host=localhost
#
# Logging
#
# Log all runtime properties on startup. Disable to avoid logging properties on startup:
druid.startup.logging.logProperties=true
#
# Zookeeper
#
druid.zk.service.host=hadoop1:2181,hadoop2:2181,hadoop3:2181
druid.zk.paths.base=/druid
#
# Metadata storage
#
# For Derby server on your Druid Coordinator (only viable in a cluster with a single Coordinator, no fail-over):
#druid.metadata.storage.type=derby
#druid.metadata.storage.connector.connectURI=jdbc:derby://localhost:1527/var/druid/metadata.db;create=true
#druid.metadata.storage.connector.host=localhost
#druid.metadata.storage.connector.port=1527
# For MySQL (make sure to include the MySQL JDBC driver on the classpath):
druid.metadata.storage.type=mysql
druid.metadata.storage.connector.connectURI=jdbc:mysql://hadoop1:3306/druid
druid.metadata.storage.connector.user=root
druid.metadata.storage.connector.password=123456
# For PostgreSQL:
#druid.metadata.storage.type=postgresql
#druid.metadata.storage.connector.connectURI=jdbc:postgresql://db.example.com:5432/druid
#druid.metadata.storage.connector.user=...
#druid.metadata.storage.connector.password=...
#
# Deep storage
#
# For local disk (only viable in a cluster if this is a network mount):
#druid.storage.type=local
#druid.storage.storageDirectory=var/druid/segments
# For HDFS:
druid.storage.type=hdfs
druid.storage.storageDirectory=hdfs://testcluster/druid/segments
# For S3:
#druid.storage.type=s3
#druid.storage.bucket=your-bucket
#druid.storage.baseKey=druid/segments
#druid.s3.accessKey=...
#druid.s3.secretKey=...
#
# Indexing service logs
#
# For local disk (only viable in a cluster if this is a network mount):
#druid.indexer.logs.type=file
#druid.indexer.logs.directory=var/druid/indexing-logs
# For HDFS:
druid.indexer.logs.type=hdfs
druid.indexer.logs.directory=hdfs://testcluster/druid/indexing-logs
# For S3:
#druid.indexer.logs.type=s3
#druid.indexer.logs.s3Bucket=your-bucket
#druid.indexer.logs.s3Prefix=druid/indexing-logs
#
# Service discovery
#
druid.selectors.indexing.serviceName=druid/overlord
druid.selectors.coordinator.serviceName=druid/coordinator
#
# Monitoring
#
druid.monitoring.monitors=["org.apache.druid.java.util.metrics.JvmMonitor"]
druid.emitter=noop
druid.emitter.logging.logLevel=info
# Storage type of double columns
# ommiting this will lead to index double as float at the storage layer
druid.indexing.doubleStorage=double
#
# Security
#
druid.server.hiddenProperties=["druid.s3.accessKey","druid.s3.secretKey","druid.metadata.storage.connector.password"]
#
# SQL
#
druid.sql.enable=true
#
# Lookups
#
druid.lookup.enableLookupSyncOnStartup=false
重點關注紅色部分，其中druid.extensions.loadList爲需要加載的外部組件配置，示例中依賴外部組件zookeeper、hadoop、mysql。（kafka依賴配置可選）
zookeeper默認配置支持zookeeper3.4.14，如何外部依賴zookeeper版本不兼容，則需要將相關的zookeeper jar拷貝到druid的lib目錄下替換原zookeeper相關jar
hadoop 默認配置支持hadoop 2.8.3客戶端，該處需要將外部依賴中hadoop的核心配置文件core-site.xml、hdfs-site.xml、mapred-site.xml、yarn-site.xml拷貝到druid中的conf/druid/cluster/_common/目錄下，如果該hadoop客戶端不兼容組件中的hadoop集羣，還需要將集羣兼容的hadoop客戶端相關jar拷貝到druid中的extensions/druid-hdfs-storage目錄下替換原來相關jar
mysql 沒有默認支持，需要將相關版本的mysql的驅動jar拷貝到extensions/mysql-metadata-storage目錄下
3.將druid整個目錄發送到hadoop2和hadoop3機器中
4.根據需要（機器配置、性能要求等）修改hadoop1中關於master節點相關配置，在Druid中的conf\druid\cluster\master\coordinator-overlord目錄下，其中runtime.properties爲druid想的屬性配置，jvm.config爲該節點進程的jvm參數配置，默認的集羣版本的jvm配置均比較高，如果用於學習測試機器配置不夠用情況下，可以將單機版的配置拷貝過來，單機版配置目錄conf\druid\single-server\micro-quickstart\coordinator-overlord
5.根據需要（機器配置、性能要求等）修改hadoop2中關於data節點相關配置，在Druid中的conf\druid\cluster\ data目錄下，該節點有兩個進程historical和middleManager，根據需要修改相關配置，類似步驟4如果測試可以選擇單機版的相關配置
6. 根據需要（機器配置、性能要求等）修改hadoop3中關於query節點相關配置，在Druid中的conf\druid\cluster\ query目錄下，該節點有兩個進程broker和router，根據需要修改相關配置，類似步驟4如果測試可以選擇單機版的相關配置
注：這裏druid集羣版本中默認配置較測試學習虛擬機要求過高，筆者測試均採用單機版配置
7.啓動
hadoop1：sh bin/start-cluster-master-no-zk-server >master.log &
hadoop2：sh bin/start-cluster-data-server > data.log &
hadoop3：sh bin/start-cluster-query-server > query.log &
8.關閉：只需要在相關的節點執行 bin/service --down 關閉當前節點的druid相關的全部進程
9.Druid相關管理界面訪問地址：
http://192.168.252.113:8888/
http://192.168.252.111:8081/
常見問題
1.關閉命令，不能使用sh bin/service --down ，因爲service 腳本獲取參數缺陷
2. bin/verify-default-ports 腳本爲啓動檢查端口占用腳本，如果你修改過相關組件的端口後，還是提示端口占用，需要關注該腳本，同步修改（該情況一般出現在單機版中）
3.如果啓動日誌中出現：Unrecogized VM option 'ExitOnOutOfMemoryError' 說明內存不足，需要修改相應進程的jvm.config配置，或者加機器內存
