ELK - 使用ElastAlert發送郵件

簡單記錄下之前的研究。

ElastAlert開源、免費，測試結果可以成功發送郵件。

https://buildmedia.readthedocs.org/media/pdf/elastalert/latest/elastalert.pdf

Install Python

之前在ElastAlert最大的坑裏說過，要用Python3！

Install Pip

如果沒有pip，則需要安裝。

sudo easy_install pip 

或
Download get-pip.py: https://bootstrap.pypa.io/get-pip.py

sudo python get-pip.py

Install Elastalert

sudo pip install elastalert

需要其他包按提示裝好。

安裝完後有這些命令可以用。

ll /usr/local/bin/ela*
/usr/local/bin/elastalert
/usr/local/bin/elastalert-create-index
/usr/local/bin/elastalert-rule-from-kibana
/usr/local/bin/elastalert-test-rule

可以這樣運行。

sudo python3 /usr/local/bin/elastalert-test-rule --config config.yaml down_frequence_rule.yaml
sudo python3 /usr/local/bin/elastalert --verbose --config config.yaml --rule down_frequence_rule.yaml
sudo python3 /usr/local/bin/elastalert --verbose --rule down_frequence_rule.yaml

或者這樣。

sudo python3 -m elastalert.elastalert --verbose --config config.yaml --rule down_frequence_rule.yaml

elastalert-test-rule的效果與elastalert --debug相似，驗證規則但不真的發送郵件。

結果

先說結果。

使用elastalert-test-rule或elastalert --debug，可以看到自己定義的郵件內容。

INFO:elastalert:Queried rule Down frequency rule from 2019-11-18 11:32 CST to 2019-11-18 11:36 CST: 54 / 54 hits
INFO:elastalert:Skipping writing to ES: {'exponent': 0, 'rule_name': 'Down frequency rule', '@timestamp': '2019-11-18T03:36:20.578458Z', 'until': '2019-11-18T03:37:20.578442Z'}
INFO:elastalert:Alert for Down frequency rule at 2019-11-18T03:35:10.755Z:
INFO:elastalert:Dear Team, mq dev is down @ 2019-11-18T03:35:10.755Z, please take action!
### Number hits: 54
> Check time: 2019-11-18T03:35:10.755Z
> IP: 192.168.1.88
> Env: mq dev
> Status: down
>>> Error: dial tcp 192.168.1.88:8888: connect: connection refused

正常運行時，提示如下。

INFO:elastalert:Sleeping for 59.999735 seconds
INFO:elastalert:Sent email to ['[email protected]']
INFO:elastalert:Ignoring match for silenced rule Down frequency rule
INFO:elastalert:Ran Down frequency rule from 2019-11-18 10:10 CST to 2019-11-18 10:25 CST: 225 query hits (212 already seen), 2 matches, 1 alerts sent

Create Index in Elasticsearch

ElastAlert的狀態保存，需要在Elasticsearch裏create index。

可以直接用示例配置來試。

cp config.yaml.example config.yaml
sudo python3 /usr/local/bin/elastalert-create-index --config config.yaml

Create Rule

創建Alert規則，主要是怎樣query，怎麼trigger，定義SMTP和信息內容等。

初次測試，要確保query的條件滿足，可以觸發郵件發送。

Email那裏，遇到一個坑：提示alerts sent，然而郵件並沒有收到，亂改一通最後加上from_addr就正常了。

# Alert when the rate of events exceeds a threshold

# (Optional)
# Elasticsearch host
es_host: localhost

# (Optional)
# Elasticsearch port
es_port: 9200

# (OptionaL) Connect with SSL to Elasticsearch
#use_ssl: True

# (Optional) basic-auth username and password for Elasticsearch
#es_username: someusername
#es_password: somepassword

# (Required)
# Rule name, must be unique
name: Down frequency rule

# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: frequency

# (Required)
# Index to search, wildcard supported
index: heartbeat-*

# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 50

# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
  hours: 4

# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
filter:
- term:
    monitor.status: "down"

# (Required)
# The alert is use when a match is found
alert:
- "email"

# (required, email specific)
# a list of email addresses to send alerts to
email:
- "[email protected]"

from_addr: "[email protected]"
smtp_host: "dummy.smtp.host"
smtp_port: 25

alert_subject: "ELK Alert - {0} Down @ {1}"
alert_subject_args:
- monitor.name
- "@timestamp"

alert_text_type: alert_text_only
alert_text: |
  Dear Team, {} is down @ {}, please take action!
  ### Number hits: {}
  > Check time: {}
  > IP: {}
  > Env: {}
  > Status: {}
  >>> Error: {}

alert_text_args:
- monitor.name
- "@timestamp"
- num_hits
- "@timestamp"
- monitor.ip
- monitor.name
- monitor.status
- error.message