最近在看夜鶯的記錄規則這部分功能實現，其中新增記錄規則之後需要遠程寫入prometheus，而對於這部分功能實現，夜鶯使用的是Go實現（可參考如下地址：https://heapdump.cn/article/5597957），由於項目使用Java開發，所以針對這部分功能，只能進行重寫。下面內容爲抽取出來的主要代碼實現，僅做記錄說明。
主要的流程如下：
1> prometheus添加啓動參數
2> 調用http請求來遠程寫，數據格式是protobuf（一種自定義的編碼格式），編碼格式是snappy（一種壓縮格式）
1.1、Prometheus啓動參數添加
針對遠程寫入Prometheus，官方文檔給出了相關說明，具體可參看如下地址：https://prometheus.io/docs/prometheus/latest/storage/，文檔中指出，遠程寫入需要在prometheus服務啓動參數中添加如下參數，然後重啓服務。
--enable-feature=remote-write-receiver
如果是使用prometheus operator管理的prometheus，則需要在spec中添加如下配置（官方文檔地址如下：https://github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/api.md#prometheusspec），同時需要注意prometheus operator的鏡像版本要高於0.56.0
enableRemoteWriteReceiver: true
例如：
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
name: k8s
namespace: monitoring
spec:
enableRemoteWriteReceiver: true
<properties>
<protobuf.version>3.23.2</protobuf.version>
</properties>
<!-- 遠程寫入prometheus依賴 -->
<!-- protobuf -->
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java</artifactId>
<version>${protobuf.version}</version>
</dependency>
<dependency>
<groupId>com.google.protobuf</groupId>
<artifactId>protobuf-java-util</artifactId>
<version>${protobuf.version}</version>
</dependency>
<!-- snappy compression -->
<dependency>
<groupId>org.xerial.snappy</groupId>
<artifactId>snappy-java</artifactId>
<version>1.1.10.1</version>
</dependency>
1.3、添加GoGoProtos、Remote、Types類
GoGoProtos、Remote、Types類這三個文件，可以通過如下項目地址（https://github.com/bprasen/remotewrite）獲取，然後添加到項目中，如下圖所示
@Slf4j
@Service("recordingRuleService")
@SuppressWarnings("all")
public class RecordingRuleServiceImpl extends ServiceImpl<RecordingRuleMapper, RecordingRule> implements RecordingRuleService {
@Resource
private PromQueryService promQueryService;
@Resource
private DatasourceFeignClient datasourceFeignClient;
/**
* 遠程寫入prometheus
*
* @param ids 記錄規則主鍵Id列表
*/
@Override
public void remoteWriteToPrometheus(List<Long> ids) {
for (Long id : ids) {
RecordingRule recordingRule = this.getById(id);
List<String> datasourceIdList = Arrays.asList(recordingRule.getDatasourceIds().split(","));
datasourceIdList.forEach(datasourceId -> {
Remote.WriteRequest.Builder writeRequestBuilder = Remote.WriteRequest.newBuilder();
Types.MetricMetadata.Builder builder = Types.MetricMetadata.newBuilder();
builder.setType(Types.MetricMetadata.MetricType.UNKNOWN);
builder.setMetricFamilyName(recordingRule.getName());
builder.setHelp("helper");
Types.MetricMetadata metricMetadata = builder.build();
writeRequestBuilder.addMetadata(metricMetadata);
handleRecordingRuleLabels(recordingRule, writeRequestBuilder, datasourceId);
try {
remoteWrite(datasourceId, writeRequestBuilder);
} catch (IOException e) {
log.error("遠程寫入prometheus出錯, 異常信息爲: {}", e.getMessage(), e);
}
});
}
}
/**
* 處理記錄規則標籤
*
* @param recordingRule 記錄規則
* @param writeRequestBuilder 寫請求構建器
* @param datasourceId 數據源id
*/
private void handleRecordingRuleLabels(RecordingRule recordingRule, Remote.WriteRequest.Builder writeRequestBuilder, String datasourceId) {
List<Types.Label> labels = new ArrayList<>();
Types.TimeSeries.Builder timeSeriesBuilder = Types.TimeSeries.newBuilder();
// 自定義標籤信息
// 設置名稱, 值爲定義的記錄規則名稱
Types.Label nameLabel = Types.Label.newBuilder().setName("__name__").setValue(recordingRule.getName()).build();
labels.add(nameLabel);
// 記錄規則中定義的附加標籤
String appendTags = recordingRule.getAppendTags();
if (StringUtils.isNotBlank(appendTags)) {
Map<String, String> tagsMap = Splitter.on(",").withKeyValueSeparator("=").split(appendTags);
for (Map.Entry<String, String> tagEntry : tagsMap.entrySet()) {
Types.Label tagLabel = Types.Label.newBuilder().setName(tagEntry.getKey()).setValue(tagEntry.getValue()).build();
labels.add(tagLabel);
}
}
// 根據記錄規則中定義的promQl語句獲取查詢數據
PromQueryData queryDataInfo = promQueryService.getQueryDataInfo(recordingRule.getPromQl(), String.valueOf(DateUtil.currentSeconds()), Integer.valueOf(datasourceId));
List<PromQueryResult> queryResultList = queryDataInfo.getResult();
queryResultList.forEach(queryResult -> {
Map<String, Object> metric = queryResult.getMetric();
for (Map.Entry<String, Object> metricEntry : metric.entrySet()) {
if (!"__name__".equals(metricEntry.getKey())) {
Types.Label metricLabel = Types.Label.newBuilder().setName(metricEntry.getKey()).setValue(String.valueOf(metricEntry.getValue())).build();
labels.add(metricLabel);
}
}
String[] resultValues = queryResult.getValue().toArray(String[]::new);
// 由於prometheus寫入的時間戳到毫秒級, 而項目中定義的時間戳到秒級, 所以這裏進行了轉換
Types.Sample sample = Types.Sample.newBuilder().setTimestamp(Long.parseLong(resultValues[0] + "000"))
.setValue(Double.parseDouble(resultValues[1])).build();
// 遠程寫入prometheus
timeSeriesBuilder.addAllLabels(labels);
timeSeriesBuilder.addSamples(sample);
writeRequestBuilder.addTimeseries(timeSeriesBuilder.build());
});
}
/**
* 遠程寫入prometheus
*
* @param datasourceId 數據源id
* @param writeRequestBuilder 寫請求構建器
*/
private void remoteWrite(String datasourceId, Remote.WriteRequest.Builder writeRequestBuilder) throws IOException {
// 將寫請求使用Snappy壓縮爲字節數組
Remote.WriteRequest writeRequest = writeRequestBuilder.build();
byte[] compressed = Snappy.compress(writeRequest.toByteArray());
// 獲取遠程寫URL
DatasourceDTO datasourceDTO = datasourceFeignClient.selectById(Integer.valueOf(datasourceId)).getData();
String url = datasourceDTO.getHttp().get("url").toString();
String remoteWriteUrl = url + "/api/v1/write";
HttpPost httpPost = new HttpPost(remoteWriteUrl);
// 添加prometheus請求頭信息, 參考go版本請求發送頭
httpPost.setHeader("Content-type", "application/x-protobuf");
httpPost.setHeader("Content-Encoding", "snappy");
httpPost.setHeader("X-Prometheus-Remote-Write-Version", "0.1.0");
//添加請求頭認證信息
String authorization = Base64.getUrlEncoder().encodeToString(("username" + ":" + "password").getBytes());
httpPost.addHeader("Authorization", "Basic " + authorization);
ByteArrayEntity byteArrayEntity = new ByteArrayEntity(compressed);
httpPost.getRequestLine();
httpPost.setEntity(byteArrayEntity);
// 添加重試機制
for (int i = 1; i <= 3; i++) {
try {
CloseableHttpResponse response = httpClient.execute(httpPost);
log.info("遠程寫入prometheus數據結果, {}", response);
break;
} catch (Exception e) {
log.error("[POST/HTTP 遠程寫入Prometheus請求信息]異常, 重試次數:{}, 請求地址:{}, 異常信息:{}", i, remoteWriteUrl, Throwables.getStackTraceAsString(e));
}
}
}
}
說明：
1> 記錄規則數據表相關字段，這裏使用的是夜鶯的表設計如下（夜鶯地址：http://flashcat.cloud/docs/content/flashcat-monitor/nightingale-v6/schema/recording_rule/）
2> 遠程寫入Prometheus成功，會返回 204 狀態碼，如下所示
3> 可能遇到的問題：
這個問題，prometheus官網也給出了大致說明，多半是與時間戳格式有關，例如，項目中的時間戳到秒級，而prometheus要求到毫秒級
官網issues地址：https://github.com/prometheus/prometheus/issues/12052
4> 參考文章