Clickhouse在Docker中的编译、安装和集群部署

Clickhouse是俄罗斯Yandex公司开源的OLAP系统，单表性能最好，最近两年发展很快。百度开源的Doris是另外一个优秀的OLAP系统，我上次记录了百度开源OLAP系统Apache Doris在Centos7.0下编译和安装，这两个系统因为源码编译和安装比较繁琐，估计很多人会卡在这一步，不利于这些系统的推广和使用。

源码编译安装

请先阅读官方文档，但是不详细
https://clickhouse.tech/docs/zh/getting_started/install/
https://clickhouse.tech/docs/zh/development/build/

Docker镜像

CK在Ubuntu下安装更加友好，可以使用官方Ubuntu的镜像Docker镜像地址

$ docker pull yandex/clickhouse-server

下载完毕后，启动镜像：

docker run --name map-clickhouse-server -it -v $HOME/ck_database:/var/lib/clickhouse -v $HOME/code/ClickHouse:/ClickHouse yandex/clickhouse-server /bin/bash
下次ps一下，找到CONTAINER ID，docker attach就进去了
docker ps
docker attach CONTAINER ID

更改源

因为官方docker镜像里面各种软件都没有安装，所以，先更改一下源，比较快，可以使用阿里云的源，命令行生成sources.list文件（因为镜像中没有VIM）

sudo mv /etc/apt/sources.list /etc/apt/sources.list.bak
sudo echo "deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse" > /etc/apt/sources.list
sudo echo "deb http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse"  >> /etc/apt/sources.list 
sudo echo "deb http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb-src http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb-src http://mirrors.aliyun.com/ubuntu/ bionic-security main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb-src http://mirrors.aliyun.com/ubuntu/ bionic-updates main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb-src http://mirrors.aliyun.com/ubuntu/ bionic-proposed main restricted universe multiverse"  >> /etc/apt/sources.list
sudo echo "deb-src http://mirrors.aliyun.com/ubuntu/ bionic-backports main restricted universe multiverse"  >> /etc/apt/sources.list

安装依赖

sudo apt-get update
sudo apt-get install git
sudo apt-get install pbuilder
sudo apt-get install debhelper
sudo apt-get install lsb-release
sudo apt-get install fakeroot
sudo apt-get install debian-archive-keyring
sudo apt-get install debian-keyring

安装编译工具

安装cmake等编译工具

sudo apt-get install git cmake ninja-build

安装PPA包，这里主要是为了安装G++9

apt-get install software-properties-common
sudo apt-get install software-properties-ommon
sudo apt-add-repository ppa:ubuntu-toolchain-r/test
如果添加PPA包失败，执行如下命令
sudo apt-get install --reinstall ca-certificates
sudo -E add-apt-repository --update ppa:ubuntu-toolchain-r/test
//update一下，然后安装gcc9和g++9
sudo apt-get update
sudo apt-get install gcc-9 g++-9

设置CC和CXX的GCC版本

vim ~/.bashrc
export CC=gcc-9
export CXX=g++-9
//souce一下生效
source ~/.bashrc

下载源码

git clone --recursive [email protected]:ClickHouse/ClickHouse.git

或者分开下载

git clone https://github.com/ClickHouse/ClickHouse.git
// 下面命令下载第三方包，如果下载失败，请删除contrib目录下对应的目录，重新执行
git submodule update --init --recursive

如果网速慢，可以下载zip包

wget https://github.com/ClickHouse/ClickHouse/archive/master.zip ./

源码编译

git clone --recursive https://github.com/ClickHouse/ClickHouse.git
cd clickhouse
git submodule update --init --recursive
mkdir build
cd build
cmake ../
ninja

编译完毕，在build/programs目录下，会生成可执行文件

[map@server build]$ ls programs/ -l
total 5219304
-rwxr-xr-x 1 root root 3387833288 Apr 22 17:38 clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-benchmark -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-client -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:39 clickhouse-compressor -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-copier -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-extract-from-config -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-format -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-local -> clickhouse
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-obfuscator -> clickhouse
-rwxr-xr-x 1 root root 1956718008 Apr 22 17:36 clickhouse-odbc-bridge
lrwxrwxrwx 1 root root         10 Apr 22 18:38 clickhouse-server -> clickhouse

配置文件在clickhouse源码目录的lprograms/server/下

[map@server ClickHouse]$ ls programs/server/*.xml -l
-rw-rw-r-- 1 lihaibo lihaibo 22942 Apr 16 14:23 programs/server/config.xml
-rw-rw-r-- 1 lihaibo lihaibo  5328 Apr 13 18:08 programs/server/users.xml

遇到的问题

如果cmake过程中提示缺少第三方包，请删除contrib下相关目录重新下载

git submodule update --init --recursive

无法下载Fuzzer

路径 /ClickHouse/contrib/grpc/third_party/bloaty/third_party/libFuzzer
想办法下载下来：https://chromium.googlesource.com/chromium/llvm-project/llvm/lib/Fuzzer/
cp ../ck_patch/Fuzzer/ -r ./contrib/grpc/third_party/bloaty/third_party/libFuzzer

找不到ZLIB
修改contrib/CMakeLists.txt，把CMAKE_DISABLE_FIND_PACKAGE_ZLIB标志位设置为0，让他从标准路径上查找

$ sudo apt-get install zlib1g
$ sudo apt-get install zlib1g.dev

找不到Z3

Could NOT find Z3: Found unsuitable version "0.0.0", but required is at least "4.7.1" (found Z3_LIBRARIES-NOTFOUND)

下载链接：https://github.com/Z3Prover/z3/releases

mkdir build
cd build
cmake -G "Unix Makefiles" ../
make -j4 # Replace 4 with an appropriate number
apt-get install python3-distutils
make install
/etc/ld.so.conf

没有atomic

Host compiler must support std::atomic!
vim /etc/ld.so.conf
/usr/lib/gcc/x86_64-linux-gnu/9/
vim /etc/profile
export C_INCLUDE_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/include/:$C_INCLUDE_PATH
export CPLUS_INCLUDE_PATH=/usr/lib/gcc/x86_64-linux-gnu/9/include/:$CPLUS_INCLUDE_PATH
如果提示缺少atomic.h文件，找到拷贝到/usr/include/中
清理一下build目录，重新cmake ../

按包安装

下载相关安装包，包都打好了，只需要自己解压执行安装脚本即可。
自己编译，需要把包、配置文件等自己整理好

可以参考这篇文章clickhouse + chproxy 集群搭建

wget https://repo.yandex.ru/clickhouse/tgz/clickhouse-common-static-20.2.1.2183.tgz
wget https://repo.yandex.ru/clickhouse/tgz/clickhouse-server-20.2.1.2183.tgz
wget https://repo.yandex.ru/clickhouse/tgz/clickhouse-client-20.2.1.2183.tgz
wget https://repo.yandex.ru/clickhouse/tgz/clickhouse-test-20.2.1.2183.tgz

下载后解压，install目录的doinst.sh安装，三个包都安装一遍

cd clickhouse-server-20.2.1.2183/install/
./doinst.sh

单机启动

日志文件 /var/log/clickhouse-server/
配置文件/etc/clickhouse-server/config.xml
配置文件/etc/clickhouse-server/user.xml
状态文件/var/lib/clickhouse/status

服务端和客户端启动

启动服务器
写个小脚本 sudo vim start_ck_server.sh

sudo -u clickhouse clickhouse-server --daemon --config-file=/etc/clickhouse-server/config.xml
sleep 3
sudo cat /var/lib/clickhouse/status

客户登陆

sudo clickhouse-client

启动时的权限问题

Clickhouse用clickhouse的用户启动，需要设置权限，可以写一个脚本，如果启动失败，可以看一下是否clickhouse用户没有目录或文件权限

sudo vim chown_clickhouse_dir.sh

sudo chown -R clickhouse /etc/clickhouse-server/
sudo chown -R clickhouse /var/log/clickhouse-server/
sudo chown -R clickhouse /var/lib/clickhouse/
sudo chown -R clickhouse /usr/bin/clickhouse
sudo chown -R clickhouse /data/clickhouse
sudo chown -R clickhouse /bin/clickhouse-server
sudo chown -R clickhouse /etc/metrika.xml

启动时的ulimit -c unlimited的问题

Poco::Exception. Code: 1000, e.code() = 0, e.displayText() = File not found: /etc/clickhouse-server/config.xml0. 0xbc7786c Poco::FileNotFoundException::FileNotFoundException(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int)  in /usr/bin/clickhouse

//设置一下
ulimit -c unlimited

集群搭建

需要把安装包拷贝到每个服务器上，这次测试

1个ZK节点
3个CK节点

Zookeeper集群配置

因为Clickhouse没有节点一致性管理能力，所以必须安装zk来管理，zk的作用有三个

节点管理
元数据管理
副本管理

参考资料
ClickHouse - 多卷存储扩大存储容量（生产环境必备）
clickhouse + chproxy 集群搭建
 clickhouse 部署+介绍

下载ZK

ZK的配置较为简单，可以部署一个ZK集群，这里因为测试，所以ZK就部署了一个节点。
下载地址https://zookeeper.apache.org/releases.html
在ZK的启动脚本中设置一下JAVA_HOME，不设置启动时会出错，也可以在~/.bashrc脚本中设置

export JAVA_HOME=sudo ./zkServer.sh start

ZK的配置文件zoo_sample.cfg拷贝一个修改一下

cat ../zookeeper/apache-zookeeper-3.6.0-bin/conf/zoo.cfg

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/tmp/zookeeper
# the port at which the clients will connect
clientPort=2989
# the maximum number of client connections.
# increase this if you need to handle more clients
#maxClientCnxns=60
#
# Be sure to read the maintenance section of the
# administrator guide before turning on autopurge.
#
# http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
#
# The number of snapshots to retain in dataDir
#autopurge.snapRetainCount=3
# Purge task interval in hours
# Set to "0" to disable auto purge feature
#autopurge.purgeInterval=1

## Metrics Providers
#
# https://prometheus.io Metrics Exporter
#metricsProvider.className=org.apache.zookeeper.metrics.prometheus.PrometheusMetricsProvider
#metricsProvider.httpPort=7000
#metricsProvider.exportJvmInfo=true

每个节点的Clickhouse的配置，配置后重启一下生效

sudo vim /etc/metrika.xml

<yandex>
<clickhouse_remote_servers>
    <perftest_3shards_1replicas>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>xx.xx.42.37</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>xx.xx.42.38</host>
                <port>9000</port>
            </replica>
        </shard>
        <shard>
            <internal_replication>true</internal_replication>
            <replica>
                <host>xx.xx.42.40</host>
                <port>9000</port>
            </replica>
        </shard>
    </perftest_3shards_1replicas>
</clickhouse_remote_servers>
<!--下面是ZK的IP和端口-->
<zookeeper-servers>
  <node index="1">
    <host>xx.xx.42.37</host>
    <port>2989</port>
  </node>
</zookeeper-servers>

<!--
<macros>
    <replica>10.198.42.37</replica>
</macros>
-->

<networks>
   <ip>::/0</ip>
</networks>

<clickhouse_compression>
<case>
  <min_part_size>10000000000</min_part_size>             
  <min_part_size_ratio>0.01</min_part_size_ratio>
  <method>lz4</method>
</case>
</clickhouse_compression>
</yandex>

查看集群情况

sudo clickhouse-client登陆

XXX : ) select * from system.clusters;
SELECT *
FROM system.clusters

┌─cluster───────────────────────────┬─shard_num─┬─shard_weight─┬─replica_num─┬─host_name────┬─host_address─┬─port─┬─is_local─┬─user────┬─default_database─┬─errors_count─┬─estimated_recovery_time─┐
│ perftest_3shards_1replicas        │         1 │            1 │           1 │ xx.xx.42.37 │ xx.xx.42.37 │ 9000 │        1 │ default │                  │            0 │                       0 │
│ perftest_3shards_1replicas        │         2 │            1 │           1 │ xx.xx.42.38 │ xx.xx.42.38 │ 9000 │        0 │ default │                  │            0 │                       0 │
│ perftest_3shards_1replicas        │         3 │            1 │           1 │ xx.xx.42.39 │ xx.xx.42.39 │ 9000 │        0 │ default │                  │            0 │                       0 │
└───────────────────────────────────┴───────────┴──────────────┴─────────────┴──────────────┴──────────────┴──────┴──────────┴─────────┴──────────────────┴──────────────┴─────────────────────────┘

然后，就可以愉快的玩耍了：）

Clickhouse在Docker中的编译、安装和集群部署

源码编译安装

Docker镜像

更改源

安装依赖

安装编译工具

下载源码

源码编译

遇到的问题

按包安装

单机启动

单机启动

服务端和客户端启动

启动时的权限问题

启动时的ulimit -c unlimited的问题

集群搭建

Zookeeper集群配置

下载ZK

查看集群情况

Wireshark 安装+使用（一）

VMware 7.1 手動模式安裝Centos 5.3

g++undefined reference可能是鏈接庫的順序不對

研發管理的歷史文章目錄

百度開源OLAP系統Apache Doris在Centos7.0下編譯和安裝

Google Python 風格指南

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結