KuduMaster遷移

KuduMaster遷移

主數量必須爲奇數。

官方文檔

1.添加Kudu Master

• 獲取現有master的uuid

  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  
  5315431da71e44a8ba9318330366563c
  

• 新增Kudu Master角色

• 停止所有Kudu進程（備份所有KuduMaster數據(也可以不備份，實施過程中Kudu會自動備份raft文件)）

• 格式化每臺新主計算機上的數據目錄，並記錄生成的UUID

  mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  

  • 第1臺新增節點執行結果

  [root@cdh02:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  
  I0717 11:31:04.182476 32125 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "5b36178fe35241e4b9ef734944c733e9"
  format_stamp: "Formatted at 2019-07-17 03:31:04 on cdh02"
  I0717 11:31:04.182989 32125 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "5b36178fe35241e4b9ef734944c733e9"
  format_stamp: "Formatted at 2019-07-17 03:31:04 on cdh02"
  I0717 11:31:04.184741 32125 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.001s
  
  [root@cdh02:/root]
  # 
  
  [root@cdh02:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  5b36178fe35241e4b9ef734944c733e9
  

  • 第2臺新增節點執行結果

  [root@cdh03:/root]
  # mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  [root@cdh03:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  I0717 11:32:10.213480 30641 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "d17c7304ff8f4a22b9c7d547dc540268"
  format_stamp: "Formatted at 2019-07-17 03:32:10 on cdh03"
  I0717 11:32:10.214143 30641 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "d17c7304ff8f4a22b9c7d547dc540268"
  format_stamp: "Formatted at 2019-07-17 03:32:10 on cdh03"
  I0717 11:32:10.216106 30641 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.002s
  
  [root@cdh03:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  d17c7304ff8f4a22b9c7d547dc540268
  
  

  • 第3臺新增節點執行結果

  # mkdir -p /kudu/master/fs_wal_dir && mkdir -p /kudu/master/fs_data_dirs && chown -R kudu:kudu /kudu/master/fs_wal_dir && chown -R kudu:kudu /kudu/master/fs_data_dirs && chmod 700 /kudu/master/fs_wal_dir && chmod 700 /kudu/master/fs_data_dirs
  sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  [root@cdh04:/root]
  # sudo -u kudu kudu fs format --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs
  I0717 11:33:01.965132 29273 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_data_dirs/instance:
  uuid: "6e61d6b48ff04201802e76be355fbaa8"
  format_stamp: "Formatted at 2019-07-17 03:33:01 on cdh04"
  I0717 11:33:01.965773 29273 fs_manager.cc:567] Generated new instance metadata in path /kudu/master/fs_wal_dir/instance:
  uuid: "6e61d6b48ff04201802e76be355fbaa8"
  format_stamp: "Formatted at 2019-07-17 03:33:01 on cdh04"
  I0717 11:33:01.967491 29273 fs_manager.cc:471] Time spent creating directory manager: real 0.002s       user 0.000s     sys 0.001s
  
  [root@cdh04:/root]
  # sudo -u kudu kudu fs dump uuid --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 2>/dev/null
  6e61d6b48ff04201802e76be355fbaa8
  

• 在現有的Kudu Master主機上執行

  sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 5315431da71e44a8ba9318330366563c:cdh01:7051 5b36178fe35241e4b9ef734944c733e9:cdh02:7051 d17c7304ff8f4a22b9c7d547dc540268:cdh03:7051 6e61d6b48ff04201802e76be355fbaa8:cdh04:7051
  
  

• CM上啓動現有Kudu Master主機

• 使用以下命令將主數據複製到每個新主服務器，在每臺新主計算機上執行。

  sudo -u kudu kudu local_replica copy_from_remote --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 cdh01:7051
  
  

• 啓動所有添加的新KuduMaster

2.刪除KuduMaster節點

• CM頁面停止所有Kudu服務

• CM刪除不需要的KuduMaster節點

• 在所有其他KuduMaster節點上執行

  sudo -u kudu kudu local_replica cmeta rewrite_raft_config --fs_wal_dir=/kudu/master/fs_wal_dir --fs_data_dirs=/kudu/master/fs_data_dirs 00000000000000000000000000000000 5b36178fe35241e4b9ef734944c733e9:cdh02:7051 d17c7304ff8f4a22b9c7d547dc540268:cdh03:7051 6e61d6b48ff04201802e76be355fbaa8:cdh04:7051
  

• 啓動所有剩餘的KuduMaster服務

• 啓動所有的KuduTabletServer服務

• 修改mysql元數據

  UPDATE TABLE_PARAMS
  SET PARAM_VALUE =
    'cdh02,cdh03,cdh04'
  WHERE PARAM_KEY = 'kudu.master_addresses'
  AND PARAM_VALUE = 'cdh01';
  

• 重啓Impala的Impala Catalog Server

• 確定服務成功啓動後刪除舊KuduMaster數據目錄

  Remove the data directories and WAL directory on the unwanted masters. This is a precaution to ensure that they cannot start up again and interfere with the new multi-master deployment.

• 檢查遷移結果

  sudo -u kudu kudu cluster ksck cdh02,cdh03,cdh04