ClouderaManager java api 有關查詢api 使用。

首先pom中加上此依賴：

    <dependency>
       <groupId>com.cloudera.api</groupId>
       <artifactId>cloudera-manager-api</artifactId>
       <version>5.14.2</version>

    <dependency>
      <groupId>ch.qos.logback</groupId>
      <artifactId>logback-classic</artifactId>
      <version>1.0.6</version>
    </dependency>

    <dependency>
      <groupId>com.alibaba</groupId>
      <artifactId>fastjson</artifactId>
      <version>1.2.8</version>
    </dependency>
 </dependency>

獲取cloudera-manager中所有集羣的信息：

public class ClouderaManagerCluster {
    static RootResourceV18 apiRoot;

    static {
        apiRoot = new ClouderaManagerClientBuilder().withHost(666.666.666.666).
                withPort(Integer.valueOf(7180))
                .withUsernamePassword(admin, admin).build().getRootV18();
    }

    private final static Logger LOGGER = LoggerFactory.getLogger(ClouderaManagerHost.class);


    public static void getAllCluster(){
        LOGGER.info("開始測試的時間爲{},**************開始測試獲取ClouderaManager集羣信息**************",Utils.getCurrentTime());
        ApiClusterList apiClusterList = apiRoot.getClustersResource().readClusters(DataView.FULL);
        LOGGER.info("ClouderaManager 共管理了{}個集羣",apiClusterList.getClusters().size());
        for(ApiCluster apiCluster : apiClusterList){
            ApiCluster apiCluster1 = apiRoot.getClustersResource().readCluster(apiCluster.getName());
            LOGGER.info("集羣名稱 {}",apiCluster1.getName());
            LOGGER.info("集羣顯示名稱 {}",apiCluster1.getDisplayName());
            LOGGER.info("CDH 版本：{}-{}",apiCluster1.getVersion(),apiCluster.getFullVersion());
            LOGGER.info("ClusterUrl {}",apiCluster1.getClusterUrl());
            LOGGER.info("HostUrl {}",apiCluster1.getHostsUrl());
            LOGGER.info("Cluster Uuid {}",apiCluster1.getUuid());
            LOGGER.info("集羣運行狀態 {}",apiCluster1.getEntityStatus());
        }
        LOGGER.info("結束測試的時間爲{},**************結束測試獲取ClouderaManager集羣信息**************",Utils.getCurrentTime());
    }

}

獲取cloudera-manager中每個集羣的每個主機的詳細信息：

public class ClouderaManagerHost {

    static RootResourceV18 apiRoot;

    static {
        apiRoot = new ClouderaManagerClientBuilder().withHost(666.666.666.666).
                withPort(Integer.valueOf(7180))
                .withUsernamePassword(admin, admin).build().getRootV18();
    }

    private final static Logger LOGGER = LoggerFactory.getLogger(ClouderaManagerHost.class);

    public static void getAllHost(){
        LOGGER.info("開始測試的時間爲{},**************開始測試集羣主機運行狀態**************",Utils.getCurrentTime());
        HostsResourceV10 hostsResourceV10 = apiRoot.getHostsResource();
        List<ApiHost> hostList = hostsResourceV10.readHosts(DataView.SUMMARY).getHosts();
        LOGGER.info("總共有 {} 臺主機組成集羣",hostList.size());
        for(ApiHost apiHost:hostList){
            LOGGER.info("---------------------------------------------");
            Host host = formatHost(hostsResourceV10.readHost(apiHost.getHostId()));
            LOGGER.info("主機Id : {}",host.getHostId());
            LOGGER.info("主機名： {}",host.getHostName());
            LOGGER.info("主機IP： {}",host.getIpAddress());
            LOGGER.info("主機線程數：{}",host.getNumCores());
            LOGGER.info("上次上報心跳時間 ：{}",host.getLastHeart());
            LOGGER.info("核心數：{}",host.getNumPhysicalCores());
            LOGGER.info("機架：{}",host.getRack());
            LOGGER.info("內存（G）：{}",host.getTotalPhysMemBytes());
            LOGGER.info("進程：{}", JSON.toJSON(host.getServices()));
            LOGGER.info("---------------------------------------------");
        }
        LOGGER.info("結束測試的時間爲{},**************結束測試集羣主機運行狀態**************",Utils.getCurrentTime());
    }

    public static Host formatHost(ApiHost apiHost){
        Host host = new Host();
        List<String> services = new ArrayList<>();
        host.setHostId(apiHost.getHostId());
        host.setHostName(apiHost.getHostname());
        host.setIpAddress(apiHost.getIpAddress());
        host.setNumCores(apiHost.getNumCores());
        host.setNumPhysicalCores(apiHost.getNumPhysicalCores());
        host.setLastHeart(apiHost.getLastHeartbeat().toString());
        host.setRack(apiHost.getRackId());
        host.setTotalPhysMemBytes(apiHost.getTotalPhysMemBytes()/1073741824);
        for(ApiRoleRef apiRoleRef:apiHost.getRoleRefs()){
            services.add(apiRoleRef.getRoleName());
        }
        host.setServices(services);
        return host;
    }
}



public class Host {
    private String hostId;

    private String hostName;

    private String ipAddress;

    private String rack;

    private String lastHeart;

    private List<String> services = new ArrayList<>();

    private long numCores;

    private long numPhysicalCores;

    private long totalPhysMemBytes;

    public void setHostId(String hostId) {
        this.hostId = hostId;
    }

    public void setHostName(String hostName) {
        this.hostName = hostName;
    }

    public void setIpAddress(String ipAddress) {
        this.ipAddress = ipAddress;
    }

    public void setRack(String rack) {
        this.rack = rack;
    }

    public void setLastHeart(String lastHeart) {
        this.lastHeart = lastHeart;
    }

    public void setServices(List<String> services) {
        this.services = services;
    }

    public void setNumCores(long numCores) {
        this.numCores = numCores;
    }

    public void setNumPhysicalCores(long numPhysicalCores) {
        this.numPhysicalCores = numPhysicalCores;
    }

    public void setTotalPhysMemBytes(long totalPhysMemBytes) {
        this.totalPhysMemBytes = totalPhysMemBytes;
    }

    public String getHostId() {
        return hostId;
    }

    public String getHostName() {
        return hostName;
    }

    public String getIpAddress() {
        return ipAddress;
    }

    public String getRack() {
        return rack;
    }

    public String getLastHeart() {
        return lastHeart;
    }

    public List<String> getServices() {
        return services;
    }

    public long getNumCores() {
        return numCores;
    }

    public long getNumPhysicalCores() {
        return numPhysicalCores;
    }

    public long getTotalPhysMemBytes() {
        return totalPhysMemBytes;
    }
}

獲取cloudera-manager上所有集羣的所有服務的狀態（這裏只是粗略的，不涉及時間點位的）：

public class ClouderaManagerService {

    private final static Logger LOGGER = LoggerFactory.getLogger(ClouderaManagerService.class);

    static RootResourceV18 apiRoot;

    static {
        apiRoot = new ClouderaManagerClientBuilder().withHost(666.666.666.666).
                withPort(Integer.valueOf(7180))
                .withUsernamePassword(admin, admin).build().getRootV18();
    }

    public static void getAllService(){
        LOGGER.info("開始測試的時間爲{},**************開始測試集羣服務運行狀態**************",Utils.getCurrentTime());
        ApiClusterList apiClusterList = apiRoot.getClustersResource().readClusters(DataView.SUMMARY);
        for(ApiCluster apiCluster:apiClusterList){
            LOGGER.info("集羣名稱：{}",apiCluster.getDisplayName());
            LOGGER.info("CDH 版本：{}-{}",apiCluster.getVersion(),apiCluster.getFullVersion());
            ServicesResourceV10 servicesResourceV10 = apiRoot.getClustersResource().getServicesResource(apiCluster.getName());
            List<ApiService> apiServices = servicesResourceV10.readServices(DataView.FULL).getServices();
            LOGGER.info("集羣總共有：{} 個service 在運行",apiServices.size());
            for(ApiService apiService:apiServices){
                Service service = formatService(apiService);
                LOGGER.info("***********************************");
                LOGGER.info("service 名稱 {}",service.getName());
                LOGGER.info("service 類型 {}",service.getType());
                for(Agent agent:service.getAgentList()) {
                    LOGGER.info("節點名稱 {}", agent.getName());
                    LOGGER.info("節點狀態 {}", agent.getStatus());
                }
                LOGGER.info("***********************************");
            }
        }
        LOGGER.info("結束測試的時間爲{},**************結束測試集羣服務運行狀態**************",Utils.getCurrentTime());
    }

    public static Service formatService(ApiService apiService){
        Service service = new Service();
        List<Agent> agents = new ArrayList<>();
        service.setName(apiService.getName());
        service.setType(apiService.getType());
        for(ApiHealthCheck apiHealthCheck:apiService.getHealthChecks()){
            Agent agent =new Agent();
            agent.setName(apiHealthCheck.getName());
            agent.setStatus(apiHealthCheck.getSummary());
            agents.add(agent);
        }
        service.setAgentList(agents);
        return service;
    }

    public static void getAllServiceRoles(){
        LOGGER.info("開始測試的時間爲{},**************開始測試集羣各個服務的roles運行狀態**************",Utils.getCurrentTime());
        ApiClusterList apiClusterList = apiRoot.getClustersResource().readClusters(DataView.SUMMARY);
        for(ApiCluster apiCluster:apiClusterList){
            LOGGER.info("集羣名稱：{}",apiCluster.getDisplayName());
            LOGGER.info("CDH 版本：{}-{}",apiCluster.getVersion(),apiCluster.getFullVersion());
            ServicesResourceV18 servicesResourceV18 = apiRoot.getClustersResource().getServicesResource(apiCluster.getName());
            List<ApiService> apiServices = servicesResourceV18.readServices(DataView.FULL).getServices();
            LOGGER.info("集羣總共有：{} 個service 在運行",apiServices.size());
            for(ApiService apiService:apiServices){
                RolesResourceV11 rolesResourceV11 = servicesResourceV18.getRolesResource(apiService.getName());
                LOGGER.info("---------------------服務名稱是{}---------------------",apiService.getName());
                for(ApiRole apiRole :rolesResourceV11.readRoles()){
                    LOGGER.info("***************************",apiRole.getName());
                    LOGGER.info("role名稱 {}",apiRole.getName());
                    LOGGER.info("role類型 {}",apiRole.getType());
                    LOGGER.info("所屬集羣 {}",apiRole.getServiceRef().getClusterName());
                    LOGGER.info("所屬服務 {}",apiRole.getServiceRef().getServiceName());
                    LOGGER.info("主機ID {}",apiRole.getHostRef().getHostId());
                    LOGGER.info("roleUrl {}",apiRole.getRoleUrl());
                    LOGGER.info("role狀態 {}",apiRole.getRoleState());
                    LOGGER.info("運行狀態總結 {}",apiRole.getHealthSummary());
                    LOGGER.info("entityStatus {}",apiRole.getEntityStatus());
                    LOGGER.info("roleConfigGroupName {}",apiRole.getRoleConfigGroupRef().getRoleConfigGroupName());
                    LOGGER.info("configStalenessStatus {}",apiRole.getConfigStalenessStatus());
                    LOGGER.info("haStatus {}",apiRole.getHaStatus());
                    for(ApiHealthCheck apiHealthCheck:apiRole.getHealthChecks()){
                        LOGGER.info("health check name {}",apiHealthCheck.getName());
                        LOGGER.info("health check summary {}",apiHealthCheck.getSummary());
                        LOGGER.info("health check suppressed {}",apiHealthCheck.getSuppressed());
                    }
                    LOGGER.info("***************************");
                }
                LOGGER.info("--------------------------------------------------------",apiService.getName());
            }
        }
        LOGGER.info("結束測試的時間爲{},**************結束測試集羣各個服務的roles運行狀態**************",Utils.getCurrentTime());
    }

}

public class Service {
    private String name;

    private String type;

    private List<Agent> agentList = new ArrayList<>();

    public void setName(String name) {
        this.name = name;
    }

    public void setType(String type) {
        this.type = type;
    }

    public void setAgentList(List<Agent> agentList) {
        this.agentList = agentList;
    }

    public String getName() {
        return name;
    }

    public String getType() {
        return type;
    }

    public List<Agent> getAgentList() {
        return agentList;
    }
}


public class Agent {
    private String name;

    private ApiHealthSummary status;

    public void setName(String name) {
        this.name = name;
    }

    public void setStatus(ApiHealthSummary status) {
        this.status = status;
    }

    public String getName() {
        return name;
    }

    public ApiHealthSummary getStatus() {
        return status;
    }
}

獲取cloudera-manager中某個服務的時間序列點位：

public class ClouderaManagerServiceMetrics {

    private final static Logger LOGGER = LoggerFactory.getLogger(ClouderaManagerServiceMetrics.class);

    static RootResourceV18 apiRoot;

    static {
        apiRoot = new ClouderaManagerClientBuilder().withHost(666.666.666.666).
                withPort(Integer.valueOf(7180))
                .withUsernamePassword(admin, admin).build().getRootV18();
    }

    public static List<Metric> formatApiTimeSeriesResponse(List<ApiTimeSeriesResponse> apiTimeSeriesResponseList){
        List<Metric> metrics = new ArrayList<>();
        for(ApiTimeSeriesResponse apiTimeSeriesResponse:apiTimeSeriesResponseList) {
            List<Data> dataList = new ArrayList<>();
            List<ApiTimeSeries> apiTimeSeriesList = apiTimeSeriesResponse.getTimeSeries();
            for (ApiTimeSeries apiTimeSeries : apiTimeSeriesList) {
                Metric metric = new Metric();
                metric.setMetricName(apiTimeSeries.getMetadata().getMetricName());
                metric.setEntityName(apiTimeSeries.getMetadata().getEntityName());
                metric.setStartTime(apiTimeSeries.getMetadata().getStartTime().toString());
                metric.setEndTime(apiTimeSeries.getMetadata().getEndTime().toString());
                for (ApiTimeSeriesData apiTimeSeriesData : apiTimeSeries.getData()) {
                    Data data = new Data();
                    data.setTimestamp(apiTimeSeriesData.getTimestamp().toString());
                    data.setType(apiTimeSeriesData.getType());
                    data.setValue(apiTimeSeriesData.getValue());
                    dataList.add(data);
                }
                metric.setData(dataList);
                metrics.add(metric);
            }
        }
        return metrics;
    }

    public static List<Metric> getServiceMetrics(String query,String startTime , String endTime){
        TimeSeriesResourceV6 timeSeriesResourceV6 = apiRoot.getTimeSeriesResource();
        String[] params = new String []{query,startTime,endTime};
        LOGGER.info("query sql is {} ,startTime is {} ,endTime is now",params);

        LOGGER.info("開始測試的時間爲{},**************開始查詢某個服務點位狀態**************",Utils.getCurrentTime());
        ApiTimeSeriesResponseList response = timeSeriesResourceV6.queryTimeSeries(query,startTime,endTime);
        List<ApiTimeSeriesResponse> apiTimeSeriesResponseList = response.getResponses();
        List<Metric> metrics = formatApiTimeSeriesResponse(apiTimeSeriesResponseList);
        LOGGER.info("查詢時間序列點位:{}", JSON.toJSON(metrics));
        LOGGER.info("結束測試的時間爲{},**************結束查詢某個服務點位狀態**************",Utils.getCurrentTime());
        return metrics;
    }

}

public class Metric {
    private String metricName;

    private String entityName;

    private String startTime;

    private String endTime;

    List<Data> data = new ArrayList<>();

    public void setMetricName(String metricName) {
        this.metricName = metricName;
    }

    public void setEntityName(String entityName) {
        this.entityName = entityName;
    }

    public void setStartTime(String startTime) {
        this.startTime = startTime;
    }

    public void setEndTime(String endTime) {
        this.endTime = endTime;
    }

    public void setData(List<Data> data) {
        this.data = data;
    }

    public String getMetricName() {
        return metricName;
    }

    public String getEntityName() {
        return entityName;
    }

    public String getStartTime() {
        return startTime;
    }

    public String getEndTime() {
        return endTime;
    }

    public List<Data> getData() {
        return data;
    }
}

public class Data {
    private String timestamp;

    private Double value;

    private String type;

    public void setTimestamp(String timestamp) {
        this.timestamp = timestamp;
    }

    public void setValue(Double value) {
        this.value = value;
    }

    public void setType(String type) {
        this.type = type;
    }

    public String getTimestamp() {
        return timestamp;
    }

    public Double getValue() {
        return value;
    }

    public String getType() {
        return type;
    }
}

這個時間點位可能比較抽象，舉個例子：

這種以時間爲橫軸，點位爲縱軸的我們統稱爲時間序列點位。

讓我們看看這個怎麼用：

ClouderaManagerServiceMetrics.getServiceMetrics("SELECT health_good_rate * 100 AS \"good health\" WHERE entityName = \"hbase\" AND category = SERVICE"
                ,"2018-05-09","now");

我們可能會說，沃日，這個sql語句我怎麼知道。。

這樣不就可以獲取sql語句了嘛。

最後貼上調用方法，放在main方法裏面，即可讓程序跑起來：

ClouderaManagerCluster.getAllCluster();

ClouderaManagerHost.getAllHost();

ClouderaManagerService.getAllService();

ClouderaManagerService.getAllServiceRoles();

ClouderaManagerServiceMetrics.getServiceMetrics("SELECT health_good_rate * 100 AS \"good health\" WHERE entityName = \"hbase\" AND category = SERVICE"
                ,"2018-05-09","now");

我自己整理了一分api簡單文檔，可以看哈~：

1。cloudera manager api 入口：
用來創建cloudera manager 的入口，需要輸入地址以及賬號密碼來創建一個根資源的實例：

RootResourceV10 apiRoot = new ClouderaManagerClientBuilder().withHost(“xxx.xxx.xxx.xx”)
.withUsernamePassword(“123”, “123”).build().getRootV10();

RootResourceV10這個對象不是必須的，它還有V9，V8等等許多，只不過越向後的版本兼容前面的並且增加了新功能。

2。獲取主機信息：
HostsResourceV10 hostsResourceV10 = apiRoot.getHostsResource();
通過cloudera manager 的入口api，獲取一個主機資源的實例，V10的意思同上。
List hostList = hostsResourceV10.readHosts(DataView.SUMMARY).getHosts();
這樣就可以得到一個apihost的一個列表對象，這裏面可能獲得的不全面，可以通過Apihost開面的hostId傳入到
hostsResourceV10.readHost(apiHost.getHostId())
當中，來獲取某個主機的信息。

3。獲取服務狀態
還是要藉助cloudera manager的公共入口RootResourceV10來進行，首先要獲取服務狀態，你要先知道你的cloudera manager 管理了幾個集羣。

ApiClusterList apiClusterList = apiRoot.getClustersResource().readClusters(DataView.SUMMARY);
ApiClusterList 是ApiCluster的列表，可以遍歷ApiCluster來獲取每個ApiCluster，ApiCluster裏面包含了集羣的詳細信息，包含CDH版本，集羣名稱等等。。

ServicesResourceV10 servicesResourceV10 = apiRoot.getClustersResource().getServicesResource(apiCluster.getName());

通過獲取的集羣名稱，你可以獲得當前集羣下的服務實例

List<ApiService>  apiServices = servicesResourceV10.readServices(DataView.FULL).getServices();

然後將所有服務資源轉化爲ApiService的一個列報表，這樣我們就可以通過遍歷ApiService來獲取每個服務的狀態了

4。獲取服務中各個主節點和從節點詳細狀態

ServicesResourceV10 servicesResourceV10 = apiRoot.getClustersResource().getServicesResource(apiCluster.getName());

通過獲取的集羣名稱，你可以獲得當前集羣下的服務實例

List<ApiService>  apiServices = servicesResourceV10.readServices(DataView.FULL).getServices();

通過遍歷獲取每個服務。

RolesResourceV11 rolesResourceV11 = servicesResourceV18.getRolesResource(apiService.getName());

然後通過RolesResourceV11獲得一個service下面得所有role，然後通過
rolesResourceV11.readRoles()
獲取一個ApiRole的鏈表。遍歷即可獲取一個service下的所有role。

5。獲取服務或者主機點位時間序列：
TimeSeriesResourceV6 timeSeriesResourceV6 = apiRoot.getTimeSeriesResource();
也是依賴於cloudera manager 的公共入口RootResourceV10，獲得一個TimeSeriesResource 用來查詢時間序列點位，
ApiTimeSeriesResponseList response = timeSeriesResourceV6.queryTimeSeries(query,startTime,endTime);
可以採用queryTimeSeries方法，來進行查詢，query是一個sql語句，然後是開始時間，結束時間，下面給出兩個示例查詢。

“SELECT health_good_rate * 100 AS \”good health\” WHERE entityName = \”hbase\” AND category = SERVICE” ,”2018-05-09”,”now”

然後獲取查詢結果

List<ApiTimeSeriesResponse> apiTimeSeriesResponseList=response.getResponses();

獲取一個ApiTimeSeriesResponse的一個列表，來獲取查詢結果即可。

好啦好啦，就介紹到這裏，溜了溜了~

ClouderaManager java api 有關查詢api 使用。

我自己整理了一分api簡單文檔，可以看哈~：

在HDP安裝的yarn基礎上，自定義安裝個spark on yarn

Spark 當中map，flatMap，mapPartitions的區別以及示例

yarn 問題總結（目前遇到的問題）

二、用IDEA導入spark源碼，並在本地運行standlone

一、Spark 2.4.0 源碼編譯

Mac下配置sublime實現LaTeX

https://yachay.unat.edu.pe/blog/index.php?comment_area=format_blog&comment_component=blog&comment_co

linux以太網驅動總結