Apache HBase HBCK2 Tool
HBCK2是hbase-1.x修復工具（ 也稱作hbck1）的繼承者。 使用它代替hbck1進行hbase-2.x安裝修復。
hbck1
不應該對hbase-2.x羣集運行hbase-1.x附帶的hbck工具（ hbck1）。 它可能會造成損壞。 雖然hbck1仍然捆綁在hbase-2.x--中以最大限度地減少意外（它在其輸出的幫助有一個指向HBCK2的提示） - 它的寫入工具（-fix）已被刪除。 它可以報告hbase-2.x羣集的狀態，但其評估可能不準確，因爲它不瞭解hbase-2.x的內部工作原理。
HBCK2比hbck1做的少得多，因爲hbck1解決的許多問題都不再是hbase-2.x中的問題，或者我們已經（或將會製作）專用工具來執行hbck1所使用的內容。 HBCK2的工作方式與hbck1的操作方式不同，要求HBase Master響應請求，而不是在hbck1工具中複製Master以外的功能。
Building HBCK2
Run:
mvn install
The built HBCK2 fat jar will be in the target sub-directory.
Running HBCK2
org.apache.hbase.HBCK2 是HBCK2的類名. 運行如下命令可以查看 HBCK2 用法:
$ HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar ./bin/hbase org.apache.hbase.HBCK2
usage: HBCK2 [OPTIONS] COMMAND <ARGS>
Options:
-d,--debug run with debug output
-h,--help output this help message
-p,--hbase.zookeeper.property.clientPort port of target hbase ensemble
-q,--hbase.zookeeper.quorum <arg> ensemble of target hbase
-v,--version this hbck2 version
-z,--zookeeper.znode.parent parent znode of target hbase
Commands:
assigns [OPTIONS] <ENCODED_REGIONNAME>...
Options:
-o,--override override ownership by another procedure
A 'raw' assign that can be used even during Master initialization.
Skirts Coprocessors. Pass one or more encoded RegionNames.
1588230740 is the hard-coded name for the hbase:meta region and
de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
encoded Region name looks like. For example:
$ HBCK2 assign 1588230740 de00010733901a05f5a2a3a382e27dd4
Returns the pid(s) of the created AssignProcedure(s) or -1 if none.
bypass [OPTIONS] <PID>...
Options:
-o,--override override if procedure is running/stuck
-r,--recursive bypass parent and its children. SLOW! EXPENSIVE!
-w,--lockWait milliseconds to wait on lock before giving up;
default=1
Pass one (or more) procedure 'pid's to skip to procedure finish.
Parent of bypassed procedure will also be skipped to the finish.
Entities will be left in an inconsistent state and will require
manual fixup. May need Master restart to clear locks still held.
Bypass fails if procedure has children. Add 'recursive' if all
you have is a parent pid to finish parent and children. This
is SLOW, and dangerous so use selectively. Does not always work.
unassigns <ENCODED_REGIONNAME>...
Options:
-o,--override override ownership by another procedure
A 'raw' unassign that can be used even during Master initialization.
Skirts Coprocessors. Pass one or more encoded RegionNames:
1588230740 is the hard-coded name for the hbase:meta region and
de00010733901a05f5a2a3a382e27dd4 is an example of what a user-space
encoded Region name looks like. For example:
$ HBCK2 unassign 1588230740 de00010733901a05f5a2a3a382e27dd4
Returns the pid(s) of the created UnassignProcedure(s) or -1 if none.
setTableState <TABLENAME> <STATE>
Possible table states: ENABLED, DISABLED, DISABLING, ENABLING
To read current table state, in the hbase shell run:
hbase> get 'hbase:meta', '<TABLENAME>', 'table:state'
A value of \x08\x00 == ENABLED, \x08\x01 == DISABLED, etc.
An example making table name 'user' ENABLED:
$ HBCK2 setTableState users ENABLED
Returns whatever the previous table state was.
HBCK2 Overview
HBCK2 is currently a simple tool that does one thing at a time only.
HBCK2 does not do diagnosis, leaving that function to other tooling, described below.
HBCK2目前是一個簡單的工具，一次只做一件事。
HBCK2不做診斷，將該功能留給其他工具， 如下面所描述的。
In hbase-2.x, the Master is the final arbiter of all state, so a general principal of HBCK2 is that it asks the Master to effect all repair. This means a Master must be up before you can run an HBCK2 command.
在hbase-2.x中，Master是所有狀態的最終仲裁者，因此HBCK2的一般原則是它要求Master執行所有修復。 這意味着在運行HBCK2命令之前必須啓動Master。
HBCK2 works by making use of an intentionally obscured HbckService hosted on the Master. The Service publishes a few methods for the HBCK2 tool to pull on. The first thing HBCK2 does is poke the cluster to ensure the service is available. It will fail if it is not or if the HbckService is lacking a wanted facility. HBCK2 versions should be able to work across multiple hbase-2 releases. It will fail with a complaint if it is unable to run. There is no HbckService in versions of hbase before 2.0.3 and 2.1.1. HBCK2 will not work against these versions.
HBCK2通過使用託管在Master上的HbckService來工作。 該服務發佈了一些HBCK2工具的方法。 HBCK2做的第一件事就是確保集羣的服務可用。 如果不可用或者HbckService缺少想要的設施，它將會失敗.HBCK2版本應該能夠在多個hbase-2版本中工作。 如果無法運行，它將提示失敗。 在2.0.3和2.1.1之前的hbase版本中沒有HbckService。 HBCK2不適用於這些版本。
發現問題
hbck1分析並報告你集羣情況是好是壞，hbck2就沒那麼自以爲是了。在hbase-2.x中，操作人確定需要修復的內容，然後使用包括hbck2在內的工具進行修復。
要發現分配中的問題，請使用以下工具
診斷工具
Master Logs
The Master runs all assignments, server crash handling, cluster start and stop, etc. In hbase-2.x, all that the Master does has been cast as Procedures run on a state machine engine. See Procedure Framework and Assignment Manager
for detail on how this new infrastructure works.
Master執行所有分配，服務器崩潰處理，集羣啓動和停止等。在hbase-2.x中，Master執行的所有操作顯示爲過程，運行在狀態機引擎上。
Each Procedure has a unique Procedure id, its pid, that it lists on each logging. Following the pid, you can trace the lifecycle of a Procedure in the Master logs as Procedures transition from start, through each of the Procedure's various stages to finish. Some Procedures spawn sub-procedures, wait on their Children, and then themselves finish. Each child logs its pid but also its ppid; its parent's pid.
每個過程都有一個唯一的過程ID，它的pid，它在每次日誌記錄中列出。你可以通過pid跟蹤Master日誌中過程的生命週期，從開始、各個階段、直至完成。 一些過程產生子過程，等待他們的子進程完成，然後自身才完成。 每個孩子都記錄了自己的pid，還記錄了ppid: 它的父母的pid。
Generally all runs problem free but if some unforeseen circumstance arises, the assignment framework may sustain damage requiring operator intervention. Below we will discuss some such scenarios but they can manifest in the Master log as a Region being STUCK or a Procedure transitioning an entity -- a Region or a Table -- may be blocked because another Procedure holds the exclusive lock and is not letting go.
通常所有運行都沒有問題，但是如果出現一些不可預見的情況，分配體系可能會發生需要操作員干預的損壞。 下面我們將討論一些這樣的場景，在Master日誌中顯示爲一個region是STUCK或一個過程轉換region或表狀態被阻止，因爲另一個過程持有獨佔鎖並且不放手。
STUCK Procedures look like this:
被阻塞的procedure如下：
2018-09-12 15:29:06,558 WARN org.apache.hadoop.hbase.master.assignment.AssignmentManager: STUCK Region-In-Transition rit=OPENING, location=va1001.example.org,22101,1536173230599, table=IntegrationTestBigLinkedList_20180626110336, region=dbdb56242f17610c46ea044f7a42895b
/master-status#tables
This section about midway down in Master UI home-page shows a list of tables with columns for whether the table is ENABLED, ENABLING, DISABLING, or DISABLED among other attributes. Also listed are columns with counts of Regions in their various transition states: OPEN, CLOSED, etc. A read of this table is good for figuring if the Regions of this table have a proper disposition. For example if a table is ENABLED and there are Regions that are not in the OPEN state and the Master Log is silent about any ongoing assigns, then something is amiss.
Master UI主頁中間顯示了一個table列表，其中包含表是否爲ENABLED，ENABLING，DISABLING或DISABLED等列。還列出了具有各種過渡狀態（如OPEN，CLOSED）的region個數。讀取此表有助於確定此表的region是否處於適當的狀態。例如，如果表是ENABLED並且存在未處於OPEN狀態的region，而Master Log 對任何正在進行的分配毫無反應，那麼肯定存在問題。
Procedures & Locks
This page off the Master UI home page under the Procedures & Locks menu item in the page heading lists all ongoing Procedures and Locks as well as the current set of Master Procedure WALs (named pv2-0000000000000000###.log under the MasterProcWALs directory in your hbase install). On startup, on a large cluster when furious assigning is afoot, this page is filled with lists of Procedures and Locks. The count of MasterProcWALs will bloat too. If after the cluster settles, there is a stuck Lock or Procedure or the count of WALs doesn't ever come down but only grows, then operator intervention is needed to alieve the blockage.
在Master UI主頁的“Procedures & Locks ”菜單下，列出了所有正在進行的過程和鎖以及當前的主過程WAL集合（在MasterProcWALs目錄下命名爲pv2-0000000000000000 ##log）。在啓動時，在大型集羣上進行激烈分配時，此頁面將填充過程和鎖定列表，MasterProcWALs數量也會膨脹。如果在羣集穩定後，存在卡住的鎖或程序，或者WAL的計數不會降低但只會增長（WAL用完會移到oldWALs），則需要操作員干預來解決堵塞問題。
Lists of locks and procedures can also be obtained via the hbase shell:
$ echo "list_locks"| hbase shell &> /tmp/locks.txt
$ echo "list_procedures"| hbase shell &> /tmp/procedures.txt
The Canary tool is useful verifying the state of assign. It can be run with a table focus or against the whole cluster.
The Canary tool 用來探查region的assign情況。
For example, to check cluster assigns:
$ hbase canary -f false -t 6000000 &>/tmp/canary.log
The -f false tells the Canary to keep going across failed Region fetches and the -t 6000000 tells the Canary run for ~two hours maximum. When done, check out /tmp/canary.log. Grep for ERROR lines to find problematic Region assigns.
You can do a probe like the Canary's in the hbase shell. For example, given a Region that has a start row of d1dddd0c belonging to the table testtable, do as follows:
實際上的原理是scan一個表中每個region的startrow起的十行。
hbase> scan 'testtable', {STARTROW => 'd1dddd0c', LIMIT => 10}
For an overview on parsing a Region name into its constituent parts, see RegionInfo API.
Other Tools
To figure the list of Regions that are not OPEN on an ENABLED or ENABLING table, read the hbase:meta table info:state column. For example, to find the state of all Regions in the table IntegrationTestBigLinkedList_20180626064758, do the following:
爲了瞭解在一個enable的table裏那些region沒打開，可以執行如下命令：
$ echo " scan 'hbase:meta', {ROWPREFIXFILTER => 'IntegrationTestBigLinkedList_20180626064758,', COLUMN => 'info:state'}"| hbase shell > /tmp/t.txt
...then grep for OPENING or CLOSING Regions.
去獲取OPENING或者CLOSING的region。
To move an OPENING issue to OPEN so it agrees with a table's ENABLED state, use the assign command in the hbase shell to queue a new Assign Procedure (watch the Master logs to see the Assign run). If many Regions to assign, use the HBCK2 tool. It can do bulk assigning.
爲了讓OPENING變爲OPEN，可以在hbaseshell中使用assign命令，或者使用HBCK2批量分配。
修復
General principals include a Region can not be assigned if it is in CLOSING state (or the inverse, unassigned if in OPENING state) without first transitioning via CLOSED: Regions must always move from CLOSED, to OPENING, to OPEN, and then to CLOSING, CLOSED.
一般原則包括如果region處於CLOSING狀態而尚未通過CLOSED，則不能assign。或者不能被uassign如果處於OPENING狀態。region必須始終按照CLOSED，OPENING，OPEN，然後再到CLOSING， CLOSED的順序變換狀態。
When making repair, do fixup a table at a time.
Also, if a table is DISABLED, you cannot assign a Region. In the Master logs, you will see that the Master will report that the assign has been skipped because the table is DISABLED. You may want to assign a Region because it is currently in the OPENING state and you want it in the CLOSED state so it agrees with the table's DISABLED state. In this situation, you may have to temporarily set the table status to ENABLED, just so you can do the assign, and then set it back again after the unassign. HBCK2 has facility to allow you do this. See the HBCK2 usage output.
此外，如果表爲DISABLED，則無法分配Region。在Master日誌中，您將看到Master將報告已跳過assign，因爲該表爲DISABLED。 您可能希望分配一個Region，因爲它當前處於OPENING狀態，並且您希望它處於CLOSED狀態，因此它與表的DISABLED狀態一致。 在這種情況下，您可能必須暫時將表狀態設置爲ENABLED，以便您可以執行分配，然後在取消分配後再次將其設置.HBCK2具有允許您執行此操作的功能。 請參閱HBCK2使用輸出。
Start-over
At an extreme, if the Master is distraught and all attempts at fixup only turn up undoable locks or Procedures that won't finish, and/or the set of MasterProcWALs is growing without bound, it is possible to wipe the Master state clean. Just move aside the /hbase/MasterProcWALs/ directory under your hbase install and restart the Master process. It will come back as a tabula rasa without memory of the bad times past.
在極端情況下，如果master一片混亂並且所有修復嘗試鎖或過程都無效，或/MasterProcWAL集合無限制地增長，則可以清理master狀態。 只需將/ hbase / MasterProcWALs /目錄移開，然後重新啓動主進程。 它將作爲一個白板而回歸，而不會記住過去的糟糕時期。
If at the time of the erasure, all Regions were happily assigned or offlined, then on Master restart, the Master should pick up and continue as though nothing happened. But if there were Regions-In-Transition at the time, then the operator may have to intervene to bring outstanding assigns/unassigns to their terminal point. Read the hbase:meta info:state columns as described above to figure what needs assigning/unassigning. Having erased all history moving aside the MasterProcWALs, none of the entities should be locked so you are free to bulk assign/unassign.
如果在清除時，所有region都被愉快地分配或離線，然後master重新啓動，master應該接收並繼續，好像什麼也沒發生。 但是如果當時存在regions-in-transition，那麼操作員可能不得不進行干預以將未完成的分配/取消分配。 閱讀如上所述的hbase：meta info：state列，以確定需要分配/取消分配的內容。 如果已MasterProcWAL的所有歷史記錄，不應有鎖定的任何實體，以便您可以自由批量分配/取消分配。
Assigning/Unassigning
Generally, on assign, the Master will persist until successful. An assign takes an exclusive lock on the Region. This precludes a concurrent assign or unassign from running. An assign against a locked Region will wait until the lock is released before making progress. See the [Procedures & Locks] section above for current list of outstanding Locks.
通常，在分配時，主人將持續到成功爲止。 assign對區域進行獨佔鎖定。 這排除了併發分配或取消分配運行。 針對鎖定區域的分配將等到鎖定被釋放後再進行。 有關未完成鎖定的當前列表，請參閱上面的[過程和鎖定]部分。
Master startup cannot progress, in holding-pattern until region onlined
meta表沒上線
This should never happen. If it does, here is what it looks like:
2018-10-01 22:07:42,792 WARN org.apache.hadoop.hbase.master.HMaster: hbase:meta,,1.1588230740 is NOT online; state={1588230740 state=CLOSING, ts=1538456302300, server=ve1017.example.org,22101,1538449648131}; ServerCrashProcedures=true. Master startup cannot progress, in holding-pattern until region onlined.
The Master is unable to continue startup because there is no Procedure to assign hbase:meta (or hbase:namespace). To inject one, use the HBCK2 tool:
HBASE_CLASSPATH_PREFIX=./hbase-hbck2-1.0.0-SNAPSHOT.jar hbase org.apache.hbase.HBCK2 assigns 1588230740
...where 1588230740 is the encoded name of the hbase:meta Region.
The same may happen to the hbase:namespace system table. Look for the encoded Region name of the hbase:namespace Region and do similar to what we did for hbase:meta.