ELK - Watcher发送告警邮件和调用接口

相比使用ElastAlert发送告警邮件，ELK提供的Wathcer要简单得多，也可以在发生警报的时候调用Web Service接口。

Configure SMTP

https://www.elastic.co/guide/en/elasticsearch/reference/current/actions-email.html

以上文档提供了多种Email系统的配置方法(elasticsearch.yml)，包括Gmail, Outlook, Microsoft Exchange, Amazon SES。

比如Gmail：

xpack.notification.email.account:
    gmail_account:
        profile: gmail
        smtp:
            auth: true
            starttls.enable: true
            host: smtp.gmail.com
            port: 587
            user: <username>

当然还要在elasticsearch-keystore里配置相应的password

bin/elasticsearch-keystore add xpack.notification.email.account.gmail_account.smtp.secure_password

实际上公司一般有内部SMTP，只需授权，而无需用户名和密码。

xpack.notification.email.account:
  work:
    profile: standard
    email_defaults:
      from: [email protected]
    smtp:
      auth: false
      starttls.enable: false
      host: my.dummy.smtp.host
      port: 25

Create Watcher

Create Threshold Alert

Management -> Elasticsearch -> Watcher -> Create threshold alert.

填入Name，Indices，Time field， 则会出现Add action按钮。

Watcher supports the following types of actions: email, webhook, index, logging, slack, and pagerduty.

Threshold Alert的主要作用，是它提供了界面，可以简单测试下配置有没有起效果。

比如Email， 填入邮件地址和内容，点击Send test email。

如果SMTP配置没问题的话，应该可以成功收到邮件。

调用接口则选Webhook，一样可以直接Send request进行测试。

注意到7.3还不支持HTTPS，7.6以后才有此选项。
Advance Watch则没有这个问题。

Create Advance Watch

以下是缺省的模板，30分钟执行一次，查询所有indices，因而一般都能执行。

把时间调小，很快就可以在elasticsearch.log里看到输出的text。

{
  "trigger": {
    "schedule": {
      "interval": "30m"
    }
  },
  "input": {
    "search": {
      "request": {
        "body": {
          "size": 0,
          "query": {
            "match_all": {}
          }
        },
        "indices": [
          "*"
        ]
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gte": 10
      }
    }
  },
  "actions": {
    "my-logging-action": {
      "logging": {
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    }
  }
}

Email Action

发送告警邮件的配置一般长这样。

  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "to": [
          "[email protected]"
        ],
        "subject": "ELK Alert - XXX is Down",
        "body": {
          "html": "{{ctx.payload.hits.total}} XXX is over limit, please take action.<p>Note: Automatic email from ELK, please do not reply."
        }
      }
    }
  }

Webhook Action

告警时调用接口配置一般长这样。

可以同时支持多个Action。

  "actions": {
    "my-logging-action": {
      "logging": {
        "level": "info",
        "text": "There are {{ctx.payload.hits.total}} documents in your index. Threshold is 10."
      }
    },
    "my-webhook-action": {
      "webhook": {
        "scheme": "https",
        "host": "my.api.dummy.host",
        "port": 8443,
        "method": "put",
        "path": "api/alert",
        "params": {},
        "headers": {
          "Content-type": "application/json"
        },
        "body": "{status: 1}"
      }
    }
  }