Debezium for PostgreSQL to Kafka 官網教程

啓動ZooKeeper

docker run -it --rm --name zookeeper -p 2181:2181 -p 2888:2888 -p 3888:3888 debezium/zookeeper:1.0

啓動Kafka

docker run -it --rm --name kafka -p 9092:9092 --link zookeeper:zookeeper debezium/kafka:1.0

啓動postgres

docker run -it --rm --name database -p 5432:5432 -e POSTGRES_PASSWORD=debezium  -d debezium/example-postgres:1.0

啓動kafka connector

docker run -it --rm --name connect -p 8083:8083 -e GROUP_ID=1 -e CONFIG_STORAGE_TOPIC=my_connect_configs -e OFFSET_STORAGE_TOPIC=my_connect_offsets -e STATUS_STORAGE_TOPIC=my_connect_statuses --link zookeeper:zookeeper --link kafka:kafka --link database:database debezium/connect:1.0

打開postman

GET: localhost:8083

{
    "version": "2.3.1",
    "commit": "18a913733fb71c01",
    "kafka_cluster_id": "sgc2CHb1TcKugWd5R6zcXA"
}

GET: http://localhost:8083/connectors/

[]

監控postgres

校驗

下方的json使用

localhost:8083/connector-plugins/io.debezium.connector.postgresql.PostgresConnector/config/validate

進行校驗，根據校驗結果進行修改

{
    "name": "inventory-connector",
    "config": {
        "name": "inventory-connector",
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "6730505c5f09",
        "database.port": "5432",
        "database.dbname": "postgres",
        "database.user": "postgres",
        "database.password": "debezium",
        "database.server.name" : "postgres"
    }
}

database.hostname： 需要從docker中查詢

connector.class： 從http://localhost:8083/connector-plugins 中查詢

database.server.name： 自定義

建立連接

Headers:

Accept:application/json

Content-Type:application/json

POST:http://localhost:8083/connectors/

{
    "name": "inventory-connector",
    "config": {
        "name": "inventory-connector",
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "tasks.max": "1",
        "database.hostname": "6730505c5f09",
        "database.port": "5432",
        "database.dbname": "postgres",
        "database.user": "postgres",
        "database.password": "debezium",
        "database.server.name" : "postgres"
    }
}

查詢

http://localhost:8083/connectors/inventory-connector

結果

{
    "name": "inventory-connector",
    "config": {
        "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
        "database.dbname": "postgres",
        "database.user": "postgres",
        "tasks.max": "1",
        "database.hostname": "6730505c5f09",
        "database.password": "debezium",
        "name": "inventory-connector",
        "database.server.name": "postgres",
        "database.port": "5432"
    },
    "tasks": [
        {
            "connector": "inventory-connector",
            "task": 0
        }
    ],
    "type": "source"
}

讓我們看一下該postgres.inventory.customers (postgres爲上方config 中的database.server.name )主題中的所有數據更改事件。我們將使用debezium/kafkaDocker映像來啓動一個新容器，該容器運行Kafka的一個實用程序來從主題開始觀看該主題：

docker run -it --name watcher --rm --link zookeeper:zookeeper --link kafka:kafka debezium/kafka:1.0 watch-topic -a -k postgres.inventory.customers

增刪查改表記錄：

可以觀察到變更記錄：