Hadoop的core-site.xml配置文件裏的fs.default.name和fs.defaultFS

在hadoop的配置文件 core-site.xml中,需要設置 fs.default.name 或 fs.defaultFS ,具體應該使用哪一個,會造什麼樣的錯誤,需要在實際操作中去驗證。官網有下面這段描述

原文 中文 core-site.xml所在目錄
This section deals with important parameters to be specified in the given configuration files 本節討論在給定的配置文件中指定的重要參數 etc/hadoop/core-site.xml
Parameter Value Notes
fs.defaultFS NameNode URI hdfs://host:port/
io.file.buffer.size 131072 Size of read/write buffer used in SequenceFiles.
We need to have only one of the two (fs.default.name or fs.defaultFS, and the former is deprecated). If both are present then NN HA will be enabled/disabled based on what is read first.

Following is description from NN HA in Apache docs:

fs.defaultFS - the default path prefix used by the Hadoop FS client when none is given.
Optionally, you may now configure the default path for Hadoop clients to use the new HA-enabled logical URI. If you used "mycluster" as the nameservice ID earlier, this will be the value of the authority portion of all of your HDFS paths. This may be configured like so, in your core-site.xml file:

<property>
    <name>fs.defaultFS</name>
    <value>hdfs://mycluster</value>
</property>

從文檔中可以看出,使用 fs.default.name 還是 使用 fs.defaultFS ,要首先判斷是否開啓了 NN 的HA (namenode 的 highavaliable),如果開啓了NN HA,那麼就用fs.defaultFS,在單一namenode的情況下,就用 fs.default.name , 如果在單一namenode節點的情況使用 fs.defaultFS ,系統將報

20/05/17 10:34:33 ERROR namenode.NameNode: Failed to start namenode.
java.lang.IllegalArgumentException: Invalid URI for NameNode address (check fs.defaultFS): file:/// has no authority.
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:626)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddressCheckLogical(DFSUtilClient.java:655)
        at org.apache.hadoop.hdfs.DFSUtilClient.getNNAddress(DFSUtilClient.java:617)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.getRpcServerAddress(NameNode.java:563)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(NameNode.java:694)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:714)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:953)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:932)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.j

hadoop查看配置的 fs.default.name名字:
hdfs getconf -confKey fs.default.name

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章