官方文檔：http://hadoop.apache.org/docs/r2.5.2/hadoop-project-dist/hadoop-hdfs/HdfsUserGuide.html#Balancer
Balancer
HDFS data might not always be be placed uniformly across the DataNode. One common reason is addition of new DataNodes to an existing cluster. While placing new blocks (data for a file is stored as a series of blocks), NameNode considers various parameters before choosing the DataNodes to receive these blocks. Some of the considerations are:
HDFS存儲在DataNode上的數據並不總是分佈均勻的。一個常見的原因就是在集羣中添加了新的DataNode節點。在存儲新的數據塊的時候，NameNode會考慮以下因素來選擇具體使用哪些DataNode來存儲新的數據塊。
- Policy to keep one of the replicas of a block on the same node as the node that is writing the block.
一份副本存放在正在寫入的節點上。- Need to spread different replicas of a block across the racks so that cluster can survive loss of whole rack.
一份副本存放在不同機架上的節點上。- One of the replicas is usually placed on the same rack as the node writing to the file so that cross-rack network I/O is reduced.
一份副本存放在與正在寫入的節點相同機架上的另一個節點上。- Spread HDFS data uniformly across the DataNodes in the cluster.
讓HDFS數據均勻的分佈在集羣的DataNode節點中。Due to multiple competing considerations, data might not be uniformly placed across the DataNodes. HDFS provides a tool for administrators that analyzes block placement and rebalanaces data across the DataNode. A brief administrator’s guide for balancer is available at HADOOP-1652.
由於一些因素，數據可能不會均勻的分佈在DataNodes節點上，HDFS提供了一個管理員命令來重新分佈和平衡DataNode上的數據塊。下面會介紹HADOOP-1652中balance遵循的原則。
HADOOP-1652：https://issues.apache.org/jira/browse/HADOOP-1652
HADOOP-1652
When a new data node joins hdfs cluster, it does not hold much data. So any map task assigned to the machine most likely does not read local data, thus increasing the use of network bandwidth. On the other hand, when some data nodes become full, new data blocks are placed on only non-full data nodes, thus reducing their read parallelism.
當集羣中添加了新的DataNode節點時，上面並沒有存放數據，所以很多map任務執行的時候就不符合數據本地化，會增加很多網絡傳輸數據的開銷。另一方面，一些數據節點存儲空間不足的時候，新的數據庫就會存放到存儲空間空閒的節點上，由此減少了並行讀。
This jira aims to find an approach to redistribute data blocks when imbalance occurs in the cluster. An solution should meet the following requirements:
當數據分佈不均衡時，Jira使用如下方式重新分佈DataNode上的數據塊，在重新分佈的時候會遵循以下幾點：
- It maintains data availablility guranteens in the sense that rebalancing does not reduce the number of replicas that a block has or the number of racks that the block resides.
需要保證數據的可用性，不減少塊的副本數和同一個機架中數據塊數量（不會跨機架移動數據塊）。- An adminstrator should be able to invoke and interrupt rebalancing from a command line.
管理員可以通過命令行的方式啓用或者終止rebalancing。- Rebalancing should be throttled so that rebalancing does not cause a namenode to be too busy to serve any incoming request or saturate the network.
rebalacing的時候不會佔用太多資源以免對namenode對外服務產生影響。
balancer使用方式
$ hdfs balancer //命令行或
$ start-balancer.sh //腳本可以使用的參數如下：
Usage: java Balancer
[-policy <policy>] the balancing policy: datanode or blockpool
datanode (default): Cluster is balanced if each datanode is balanced.
blockpool: Cluster is balanced if each block pool in each datanode is balanced.
[-threshold <threshold>] Percentage of disk capacity（默認10.0）
[-exclude [-f <hosts-file> | comma-sperated list of hosts]] Excludes the specified datanodes.
[-include [-f <hosts-file> | comma-sperated list of hosts]] Includes only the specified datanodes.在使用balancer的時候默認帶寬是有限制的（默認1MB），可以使用如下命令提高balancer可用帶寬：
$ hdfs dfsadmin [-setBalancerBandwidth <bandwidth in bytes per second>]該參數在hdfs-site.xml中默認配置如下：
<property>
<name>dfs.datanode.balance.bandwidthPerSec</name>
<value>1048576</value>
<description>
Specifies the maximum amount of bandwidth that each datanode
can utilize for the balancing purpose in term of
the number of bytes per second.
</description>
</property>