今早splunk搜索服務器出現了異常，本來轉發器有4個，上班發現只能搜索到2個，很奇怪。後來別人說轉發器可能有限制，但經過分析，轉發器是沒有限制的，找到兩個問題點
1:查找splunk日誌，發現有吞吐量限制，默認爲256
cat /opt/splunkforwarder/var/log/splunk/splunkd.log |grep limits
09-03-2014 10:59:48.466 +0800 WARN FileTracker - migrating maxDataSize value=500 from _thefishbucket in indexes.conf to limits.conf stanza=inputproc setting=file_tracking_db_threshold_mb
09-03-2014 11:05:30.726 +0800 INFO ThruputProcessor - Current data throughput (258 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:10:30.735 +0800 INFO ThruputProcessor - Current data throughput (261 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:15:30.980 +0800 INFO ThruputProcessor - Current data throughput (284 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
09-03-2014 11:20:31.230 +0800 INFO ThruputProcessor - Current data throughput (258 kb/s) has reached maxKBps. As a result, data forwarding may be throttled. Consider increasing the value of maxKBps in limits.conf.
vim /opt/splunkforwarder/etc/apps/SplunkUniversalForwarder/default/limits.conf
# Version 6.1.3
[thruput]
maxKBps = 600
之後修改了下吞吐量爲600Kb/s，問題依舊
2：經查看，是我的日誌程序進程死掉，導致日誌文件0KB，splunk搜索不到是正常的啊（solunk默認是會忽略0kb的文件哈），我改了下日誌文件，這時候就可以正常寫入日誌了，splunk會立馬搜索到數據的，呵呵