如何實現Windows,Mac,Linux訪問Hadoop HDFS 文件系統

最近有個需求需要實現windows 上直接訪問Hadoop HDFS 文件系統，並設置相關權限，

前提條件:

1.hadoop 健康集羣一個
2.centos7 系統一個
3. hadoop-fuse-dfs 安裝軟件
4. samba4 安裝軟件

第一步: 安裝hadoop-fuse-dfs

使用root用戶登陸centos7

修Ccentos-base.repo
添加以下內容到文件最後：
[cloudera-cdh5]
# Packages for Cloudera's Distribution for Hadoop, Version 5, on RedHat or CentOS 6 x86_64

name=Cloudera's Distribution for Hadoop, Version 5

baseurl=https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/5/

gpgkey =https://archive.cloudera.com/cdh5/redhat/6/x86_64/cdh/RPM-GPG-KEY-cloudera

gpgcheck = 1

然後執行:
yum install -y hadoop-fuse-dfs
yum install -y samba

如果下載慢,可以到國外服務器上下載，然後拷貝回本地安裝。
yum install -y --downloadonly --downloaddir=/tmp/hadoop-fuse-dfs hadoop-fuse-dfs
tar -czf hadoop-fuse-dfs.tgz /tmp/hadoop-fuse-dfs

yum install -y --downloadonly --downloaddir=/tmp/samba samba
tar -czf samba.tgz /tmp/samba

echo “或者從下面2個連接下載 http://nexus.nps.xszq.vip/samba.tgz . http://nexus.nps.xszq.vip/hadoop-hdfs-fuse.tgz”

scp hadoop-fuse-dfs.tgz 到目標機器
scp samba.tgz 到目標機器
或wget http://nexus.nps.xszq.vip/samba.tgz
wget http://nexus.nps.xszq.vip/hadoop-hdfs-fuse.tgz

cd /root
tar -zxvf hadoop-hdfs-fuse.tgz
cd hadoop-hdfs-fuse
yum -y localinstall *.rpm

cd /root/
tar -zxvf samba.tgz
cd samba
yum -y localinstaall *.rpm

echo “請檢查是非安裝成功.”

mkdir -p /var/server/share
chmod 777 /var/server/share
echo “執行掛載hdfs”

echo "10.6.1.13:8020 是active namenode "
hadoop-fuse-dfs -o allow_other,usetrash,nonempty dfs://10.6.1.13:8020 /mnt/hdfs
ls -al /var/server/share

#執行samba配置,把/var/server/share共享出去.
請查看samb 相關配置

查看samba的版本
#smbd --version
#Version 4.9.1

/etc/samba/smb.conf 內容如下

[root@localhost ~]# more /etc/samba/smb.conf
# See smb.conf.example for a more detailed config file or
# read the smb.conf manpage.
# Run 'testparm' to verify the config is correct after
# you modified it.


[global]
	workgroup = SAMBA
	security = user
        socket options = TCP_NODELAY IPTOS_LOWDELAY SO_RCVBUF=65536 SO_SNDBUF=65536
	passdb backend = tdbsam

	printing = cups
	printcap name = cups
	load printers = yes
	cups options = raw
        log level = 3 passdb:5 auth:5

[Anonymous]
        path=/var/server/nginx/download.cityworks.cn
        read  only = no
        guest ok   = yes

[hdfs]
        comment = HDFS
        path = /mnt/hdfs
        public = yes
        writable = yes
        printable = no
        create mask = 0744
        valid users = nobody hadoop hdfs centos
        force user = hadoop
        force group = hadoop
        available = yes
        browseable = yes
        workgroup = WORKGROUP
        display charset = UTF-8
        unix charset = UTF-8

[Share]
  guest ok   = yes
  public = yes
  path=/var/server/share
  writable = yes
  valid users = centos
  create mask = 0700
  directory mask = 0700
  force user = nobody
  force group = nogroup
  available = yes
  browseable = yes
  workgroup = WORKGROUP
  display charset = UTF-8
  unix charset = UTF-8
  dos charset = cp936


[secured]
         path = /var/server/nginx.cityworks.cn
         valid users = @smbgrp
	 read only = No

[homes]
	comment = Home Directories
	valid users = %S, %D%w%S
	browseable = No
	read only = No
	inherit acls = Yes

[printers]
	comment = All Printers
	path = /var/tmp
	printable = Yes
	create mask = 0600
	browseable = No

[print$]
	comment = Printer Drivers
	path = /var/lib/samba/drivers
	write list = root
	create mask = 0664
	directory mask = 0775

#創建一個本地賬號hadoop,並設置密碼
[root@localhost ~]# useradd hadoop
passwd hadoop

#設置smb用戶 hadoop的密碼
smbpasswd hadoop

這個2個密碼要設置一樣.

關閉selinux
#setenforce 0

然後重新啓動smb 服務
systemctl enable smb.service
systemctl enable nmb.service

systemctl restart smb.service
systemctl restart nmb.service

samba 需要打開對於的網絡端口
firewall-cmd --permanent --zone=public --add-service=samba
firewall-cmd --reload

關於samba 的安裝 可以使用ansible 自動安裝
https://github.com/HappyFreeAngel/ansible-samba.git

測試：使用window10 打開 \192.168.2.47
會提示輸入用戶和密碼 輸入 hadoop 和你的剛纔設置的密碼

打開hdfs 文件夾

經測試工作正常。
這裏可以直接拷貝或創建文件到 hdfs文件夾，馬上hadoop 裏就可以看到了。

如果要永久關閉selinux 並立即生效，執行下面代碼.
echo “永久關閉selinux”
sed -i ‘s/SELINUX=enforcing/SELINUX=disabled/g’ /etc/sysconfig/selinux
setenforce 0

如果希望系統重新啓動,還可以正常訪問，則修改/etc/fstab 內容
增加一行
hadoop-fuse-dfs#dfs://10.6.1.13:8020 /mnt/hdfs fuse allow_other,nonempty,usetrash,rw 0 0

記得把IP改爲你的hadoop IP 地址

坑: selinux 必須關閉，否則會導致無法瀏覽.

[root@localhost ~]# setenforce 0
[root@localhost ~]# service smb restart
Redirecting to /bin/systemctl restart smb.service

Mac 電腦訪問方式:
1.隨便打開一個文件夾

2.點擊 最上面的 “前往“ ， 然後鼠標移動到 彈出菜單的 最下面“連接服務器…" 點擊 “連接服務器…"

3. 在彈出的對話框內輸入你的centos7 地址 smb://你的IP地址 然後點擊連接即可。