宜人貸蜂巢ELK Stack之elasticsearch權限探索

上文宜人貸蜂巢API網關技術解密之Netty使用實踐提到了，API網關“承外對內”，將外部請求，轉發到內部各個抓取服務。在網關中，不僅可以做鑑權、加解密、路由、限流功能；如果想了解各服務接口的調用次數、調用時間、成功次數、失敗次數等接口相關的統計，都可以網關中埋點，採集相關信息，通過logstash放入elasticsearch（簡稱es），在kibana中或通過es的http請求獲得各服務接口的統計信息。

ELK框架

最近有一個需求，我們的es需要作爲一個服務提供出來。考慮到es內保存着敏感數據，需要對es添加權限管理功能。搜尋es權限管理，大概有三套方案：

• ELK Stack官方安全插件X-Pack Securit
  
• es安全插件Search Guard
• 自實現權限管理
  

考慮到X-Pack Security是收費軟件，自開發一個權限管理模塊成本比較高，最後選用了Search Guard的社區版本，其支持的功能已經滿足當前需求。

Search Guard安裝過程

• Stop Elasticsearch
  

  ps -ef | grep elasticsearch | grep -v grep | awk {'print $2'} | xargs kill -9

• Install Search Guard
  

  /opt/soft/elasticsearch-5.5.1-guard/bin/elasticsearch-plugin install -b com.floragunn:search-guard-5:5.5.1-14

  如果線上服務器不能訪問外網，可以事先下載對應版本的 Search Guard，再以本地模式安裝
  

  /opt/soft/elasticsearch-5.5.1-guard/bin/elasticsearch-plugin install -b file:///opt/soft/search-guard-5-5.5.1-15.zip

• 產生證書及配置es
  
  • 如果在測試環境安裝es，可以使用軟件自帶的腳本，生成CA及相應證書，並自動配置了es
    

    sh /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/tools/install_demo_configuration.sh

    但以上方式不適用於線上環境，我們的es希望在線上環境能提供權限訪問控制，所以採用了search-guard-ssl生成線上證書
    
  • 首先，下載了search-guard-ssl項目
    

    git clone https://github.com/floragunncom/search-guard-ssl.git

    生成證書，可以參考官方文檔詳細說明
    
  • 建議修改證書信息及密碼
    example-pki-scripts/etc/root-ca.conf
    example-pki-scripts/etc/signing-ca.conf
    
  • 最後，執行證書生成腳本
    

    sh ./example.sh

  • 將相應的證書拷貝到es的config文件夾下
    
  • 修改es配置
    

    cluster.name: honeycomb-es-guard-5.5.1
    # node 1 to 5
    node.name: node-1 
    path.data: /data01/elasticsearch-guard
    path.logs: /opt/logs/elasticsearch-guard
    
    network.host: 0.0.0.0
    
    http.cors.enabled: true
    http.cors.allow-origin: "*"
    thread_pool.bulk.queue_size: 50
    
    searchguard.ssl.transport.keystore_filepath: node-1-keystore.jks
    searchguard.ssl.transport.keystore_password: node_es
    searchguard.ssl.transport.truststore_filepath: truststore.jks
    searchguard.ssl.transport.truststore_password: r_ca_honeycomb
    searchguard.ssl.transport.enforce_hostname_verification: false
    searchguard.ssl.transport.enable_openssl_if_available: true
    searchguard.ssl.transport.resolve_hostname: false
    
    searchguard.ssl.http.enabled: true
    searchguard.ssl.http.keystore_filepath: node-1-keystore.jks
    searchguard.ssl.http.keystore_password: node_es
    searchguard.ssl.http.truststore_filepath: truststore.jks
    searchguard.ssl.http.truststore_password: r_ca_honeycomb
    
    searchguard.authcz.admin_dn:
      - CN=*,OU=client,O=client,L=test, C=de
      - CN=kirk,OU=client,O=client,L=test,C=DE
    searchguard.ssl.http.clientauth_mode: NONE

• Restart Elasticsearch.
  

  su elk
  /opt/soft/elasticsearch-5.5.1-guard/bin/elasticsearch -d

• Initialise the Search Guard index by running sgadmin
  

  /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/tools/sgadmin.sh -cd /opt/soft/elasticsearch-5.5.1-guard/plugins/search-guard-5/sgconfig/ -cn honeycomb-es-guard-5.5.1 -ks /opt/soft/elasticsearch-5.5.1-guard/config/kirk-keystore.jks -kspass ki_ca_1 -ts /opt/soft/elasticsearch-5.5.1-guard/config/truststore.jks -tspass r_ca_honeycomb –nhnv
  

• 測試結果
  

  curl -k -u admin:admin 'https://localhost:9200/_searchguard/authinfo?pretty'

es裝上Search Guard後的ELK框架

kibana修改步驟

• 安裝插件

  /opt/soft/kibana-5.5.1-linux-x86_64-guard/bin/kibana-plugin install file:///opt/soft/searchguard-kibana-5.5.1-4.zip

• kibana配置
  

  server.port: 5601
  server.host: "0.0.0.0"
  elasticsearch.url: "https://localhost:9200"
  kibana.index: ".kibana"
  
  elasticsearch.username: "kibanaserver"
  elasticsearch.password: "kibanaserver"
  elasticsearch.ssl.verificationMode: none

• start kibana
  

  /opt/soft/kibana-5.5.1-linux-x86_64-guard/bin/kibana &

• 4、訪問頁面

logstash修改步驟

logstash在按照官方文檔配置，且諮詢Search Guard官方後，並沒有安裝成功；現列出操作過程及問題現象，如有哪位讀者瞭解，還望不吝賜教

• Configure logstash to use HTTPS instead of HTTP.
  例如，hosts => "https://..."
  
• If you want to verify the server’s certificate (optional, but recommended), you need to provide the path to the keystore containing your Root CA as well.
  logstash不需要驗證es的證書，所以設置ssl_certificate_verification => false
  如果進行證書驗證，需配置CA路徑及密碼
  
• Configure the logstash username and password.
  最終配置
  

  output {
    elasticsearch {
          hosts => ["https://xx.xx.xx.32:9200"]
          index => "%{log_project}-%{+YYYY-MM-dd}"
  
          ssl => true
          ssl_certificate_verification => false
          #truststore => "/opt/soft/elasticsearch-5.5.1-guard/config/truststore.jks"
          #truststore_password => r_ca_honeycomb
  
          user => logstash
          password => logstash
    }
  }

• start logstash
  

  /opt/soft/logstash-2.4.1-guard/bin/logstash -f /opt/soft/logstash-2.4.1-guard/conf/honeycomb-logstash.conf

問題

1. Search Guard 建議logstash不做證書驗證且不需要提供truststore和truststore_password，但按操作後，進程啓動成功，但仍有fail提示

** WARNING ** Detected UNSAFE options in elasticsearch output configuration!
** WARNING ** You have enabled encryption but DISABLED certificate verification.
** WARNING ** To make sure your data is secure change :ssl_certificate_verification to true {:level=>:warn}
PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target {:class=>"Manticore::ClientProtocolException", :level=>:error}
Pipeline main started

2. logstash在啓動後，發送數據給es時，始終報錯
Attempted to send a bulk request to Elasticsearch configured at '["https://xx.xx.xx.32:9200"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided? {:error_message=>"PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target", :error_class=>"Manticore::ClientProtocolException", :backtrace=>["/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:37:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:79:in `call'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:256:in `call_once'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/manticore-0.6.0-java/lib/manticore/response.rb:153:in `code'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:84:in `perform_request'", "org/jruby/RubyProc.java:281:in `call'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/base.rb:257:in `perform_request'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/transport/http/manticore.rb:67:in `perform_request'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-transport-1.1.0/lib/elasticsearch/transport/client.rb:128:in `perform_request'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/elasticsearch-api-1.1.0/lib/elasticsearch/api/actions/bulk.rb:93:in `bulk'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:53:in `non_threadsafe_bulk'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "org/jruby/ext/thread/Mutex.java:149:in `synchronize'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:38:in `bulk'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:172:in `safe_bulk'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:101:in `submit'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:86:in `retrying_submit'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:29:in `multi_receive'", "org/jruby/RubyArray.java:1653:in `each_slice'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-output-elasticsearch-2.7.1-java/lib/logstash/outputs/elasticsearch/common.rb:28:in `multi_receive'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:130:in `worker_multi_receive'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/output_delegator.rb:114:in `multi_receive'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:301:in `output_batch'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:232:in `worker_loop'", "/opt/soft/logstash-2.4.1-guard/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.1-java/lib/logstash/pipeline.rb:201:in `start_workers'"], :level=>:error}

看日誌似乎是目的不可達，但在logstash機器上，通過命令測試，是可以訪問的

curl -k -u logstash:logstash 'https://xx.xx.xx.32:9200/_searchguard/authinfo?pretty'

另外es有打印錯誤日誌

[2017-08-24T10:39:20,834][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
[2017-08-24T10:39:21,663][ERROR][c.f.s.h.SearchGuardHttpServerTransport] [node-2] SSL Problem Received fatal alert: certificate_unknown
javax.net.ssl.SSLException: Received fatal alert: certificate_unknown
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[netty-handler-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]

3. 上文es的錯誤日誌，顯示es不知道證書（應該是客戶端證書），但是已經配置了不做客戶端認證，這又是爲什麼？

es部分配置

searchguard.ssl.http.clientauth_mode: NONE

總結

雖然logstash與es通信暫時沒有成功，但是在嘗試開通es權限的過程中，也積累了些經驗和成果
1. 使用類似文中curl的方式（http請求），直接使用es服務
2. 添加了Search Guard的es如何對外提供域名訪問，實現後端多點服務/路由
nginx可以處理來自外部的https請求，但目前不支持https代理，即nginx不能訪問後端的https服務。如果仍然使用Search Guard，可以考慮使用四層負載均衡器lvs或者硬件負載均衡器，將請求透傳給後端服務器；另外知乎文章上有推薦使用Squid。

參考

https://floragunn.com/

https://github.com/floragunncom/search-guard-docs

https://github.com/floragunncom/search-guard-docs/blob/master/installation.md

https://github.com/floragunncom/search-guard-docs/blob/master/tls_generate_demo_certificates.md

https://github.com/floragunncom/search-guard-ssl/tree/5.5.0

https://github.com/floragunncom/search-guard-docs/blob/master/kibana.md

https://github.com/floragunncom/search-guard-docs/blob/master/logstash.md

https://forum.nginx.org/read.php?2,15124,15256

https://www.zhihu.com/question/19871146