ELK - Watcher發送告警郵件和調用接口

相比使用ElastAlert發送告警郵件，ELK提供的Wathcer要簡單得多，也可以在發生警報的時候調用Web Service接口。

Configure SMTP

https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html

以上文檔提供了多種Email系統的配置方法(elasticsearch.yml)，包括Gmail, Outlook, Microsoft Exchange, Amazon SES。

比如Gmail：

xpack.notification.email.account:
    gmail_account:
        profile: gmail
        smtp:
            auth: true
            starttls.enable: true
            host: smtp.gmail.com
            port: 587
            user: <username>

當然還要在elasticsearch-keystore裏配置相應的password

bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password

實際上公司一般有內部SMTP，只需授權，而無需用戶名和密碼。

xpack.notification.email.account:
  work:
    profile: standard
    email_defaults:
      from: [email protected]
    smtp:
      auth: false
      starttls.enable: false
      host: my.dummy.smtp.host
      port: 25

Create Watcher

Create Threshold Alert

Management -> Elasticsearch -> Watcher -> Create threshold alert.

填入Name，Indices，Time field， 則會出現Add action按鈕。

Watcher supports the following types of actions: email, webhook, index, logging, slack, and pagerduty.

Threshold Alert的主要作用，是它提供了界面，可以簡單測試下配置有沒有起效果。

比如Email， 填入郵件地址和內容，點擊Send test email。

如果SMTP配置沒問題的話，應該可以成功收到郵件。

調用接口則選Webhook，一樣可以直接Send request進行測試。

注意到7.3還不支持HTTPS，7.6以後纔有此選項。
Advance Watch則沒有這個問題。

Create Advance Watch

以下是缺省的模板，30分鐘執行一次，查詢所有indices，因而一般都能執行。

把時間調小，很快就可以在elasticsearch.log裏看到輸出的text。

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 10
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

Email Action

發送告警郵件的配置一般長這樣。

  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "[email protected]"
        ],
        "subject": "ELK Alert - XXX is Down",
        "body": {
          "html": "{{ctx.payload.hits.total}} XXX is over limit, please take action.<p>Note: Automatic email from ELK, please do not reply."
        }
      }
    }
  }

Webhook Action

告警時調用接口配置一般長這樣。

可以同時支持多個Action。

  "actions": {
    "my-logging-action": {
      "logging": {
        "level": "info",
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    },
    "my-webhook-action": {
      "webhook": {
        "scheme": "https",
        "host": "my.api.dummy.host",
        "port": 8443,
        "method": "put",
        "path": "api/alert",
        "params": {},
        "headers": {
          "Content-type": "application/json"
        },
        "body": "{status: 1}"
      }
    }
  }