標籤
PostgreSQL , IoT , 軌跡聚合 , pipelinedb , 流計算 , 實時聚合
背景
IoT場景，車聯網場景，共享單車場景，人的行爲位點等，終端實時上報的是孤立的位點，我們需要將其補齊成軌跡。
例如共享單車，下單，開鎖，生成訂單，騎行，關閉訂單，關鎖。這個過程有一個唯一的訂單號，每次上報的位點會包含時間，訂單號，位置。
根據訂單號，將點聚合爲軌跡。
使用pipelinedb插件，可以實時的實現聚合。
例子
以ECS (centos 7.x x64), postgresql 10 爲例
1、編譯zeromq
wget https://github.com/zeromq/libzmq/releases/download/v4.2.5/zeromq-4.2.5.tar.gz
tar -zxvf zeromq-4.2.5.tar.gz
cd zeromq-4.2.5
./configure
make
make install
2、編譯pipelinedb
wget https://github.com/pipelinedb/pipelinedb/archive/1.0.0rev4.tar.gz
tar -zxvf 1.0.0rev4.tar.gz
cd pipelinedb-1.0.0rev4/
vi Makefile
SHLIB_LINK += /usr/local/lib/libzmq.so -lstdc++
. /var/lib/pgsql/env.sh 1925
USE_PGXS=1 make
USE_PGXS=1 make install
3、配置postgresql.conf
max_worker_processes = 512
shared_preload_libraries = 'pipelinedb'
pipelinedb.stream_insert_level=async
pipelinedb.num_combiners=8
pipelinedb.num_workers=16
pipelinedb.num_queues=16
pipelinedb.fillfactor=75
pipelinedb.continuous_queries_enabled=true
重啓
pg_ctl restart -m fast
4、安裝插件
postgres=# create extension pipelinedb;
5、創建stream，實時寫入軌跡點
CREATE FOREIGN TABLE s1 ( order_id int8, ts timestamp, pos geometry )
SERVER pipelinedb;
6、創建Continue view，實時聚合
CREATE VIEW cv1 WITH (action=materialize ) AS
select order_id, min(ts) min_ts, array_agg(ts||','||st_astext(pos)) as seg
from s1
group by order_id;
激活視圖(默認已激活)
select pipelinedb.activate('public.cv1');
7、壓測
vi test.sql
\set order_id random(1,100000)
\set x random(70,90)
\set y random(120,125)
insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random()));
pgbench -M prepared -n -r -P 1 -f ./test.sql -c 256 -j 256 -T 120
8、壓測結果
transaction type: ./test.sql
scaling factor: 1
query mode: prepared
number of clients: 256
number of threads: 256
duration: 120 s
number of transactions actually processed: 17614607
latency average = 1.740 ms
latency stddev = 1.730 ms
tps = 146550.933776 (including connections establishing)
tps = 146906.482277 (excluding connections establishing)
script statistics:
- statement latencies in milliseconds:
0.002 \set order_id random(1,10000000)
0.001 \set x random(70,90)
0.000 \set y random(120,125)
1.742 insert into s1 (order_id, ts, pos) values (:order_id, clock_timestamp(), st_makepoint(:x+10*random(), :y+10*random()));
postgres=# \x
Expanded display is on.
-[ RECORD 17 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 8672585
min_ts | 2018-11-01 18:44:08.140027
seg | {"2018-11-01 18:44:08.140027,POINT(78.3615547642112 121.881739947945)","2018-11-01 18:44:11.739248,POINT(80.9645632216707 121.450987955555)"}
-[ RECORD 18 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 4011211
min_ts | 2018-11-01 18:44:08.166407
seg | {"2018-11-01 18:44:08.166407,POINT(87.126777020283 132.819293198176)","2018-11-01 18:44:11.524995,POINT(80.482944605872 126.906906872056)"}
-[ RECORD 19 ]---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
order_id | 2468486
min_ts | 2018-11-01 18:44:08.135136
seg | {"2018-11-01 18:44:08.135136,POINT(84.7732630362734 132.659516767599)","2018-11-01 18:44:20.603312,POINT(87.6352122295648 132.18647258915)","2018-11-01 18:44:19.447776,POINT(94.9817024609074 131.295661441982)"}
9、歷史軌跡的保留
設置cv生命週期，自動清理老化數據
postgres=# select pipelinedb.set_ttl('cv1', interval '1 hour' , 'min_ts');
-[ RECORD 1 ]-----
set_ttl | (3600,2)
創建目標持久化表
create table cv1_persist (like cv1);
創建時間字段索引（CV1）
postgres=# create index idx_1 on cv1 (min_ts);
CREATE INDEX
ETL形式，將數據從cv抽取到目標持久化表
postgres=# insert into cv1_persist select * from cv1 where min_ts <= '2018-01-01';
INSERT 0 0
參考
https://github.com/pipelinedb/pipelinedb