ubuntu執行遠程腳本,分佈式系統搭建

多數執行遠程腳本都是基於ssh的,
所以
1.配置ssh免密登陸(多數分佈式系統,如hadoop)
2.執行遠程腳本

一. 配置ssh免密登陸

首先生成 Master 的公匙,在 Master 節點終端中執行:

$ cd ~/.ssh            # 如果沒有該目錄,先執行一次ssh localhost
$ ssh-keygen -t rsa    # 一直按回車就可以,生成的密鑰保存爲.ssh/id_rsa

Master 節點需能無密碼 ssh 本機,這一步還是在 Master 節點上執行:

$ cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

完成後可以使用 ssh Master 驗證一下。接着將公匙傳輸到 Slave1 節點:

$ scp ~/.ssh/id_rsa.pub hadoop@Slave1:/home/hadoop/

scp時會要求輸入Slave1上hadoop用戶的密碼(hadoop),輸入完成後會提示傳輸完畢。

接着在 Slave1節點 上將ssh公匙保存到相應位置,執行

$ cat ~/id_rsa.pub >> ~/.ssh/authorized_keys

如果有其他 Slave 節點,也要執行 將公匙傳輸到 Slave 節點、在 Slave 節點上加入授權 這兩步。

最後在 Master 節點上就可以無密碼SSH到Slave1節點了。

$ ssh Slave1

二. 執行遠程腳本

1.python+paramiko
網上的評價都是paramiko很強大
使用方式(一般的測試腳本都能跑的很順):

import paramiko
import select
client = paramiko.SSHClient()
client.load_system_host_keys()
client.connect('host.example.com')
transport = client.get_transport()
channel = transport.open_session()
channel.exec_command("tail -f /var/log/everything/current")
while True:
  rl, wl, xl = select.select([channel],[],[],0.0)
  if len(rl) > 0:
      # Must be stdout
      print channel.recv(1024)

其實我們的需求很簡單,分佈式執行JOB,分散cpu和內存的壓力,但是用paramiko的時候,總是在遠程代碼執行到20~50min後,遠程的腳本調用的進程被掛起,一直sleep(我們的遠程腳本有調用slaver本地的),不知道怎麼回事(現在也不明白,描述起來比較玄奧,有可能是這個模塊的BUG)

2.python+parallel

Linux系統映像規模當超過30時,小小的變更也會浪費很多時間,Linux系統下有很多可用的集中管理方案,例如著名的puppet,這個是用ruby開發,號稱可以管理以萬計的服務器。不過對於大多數SA來說,這個不怎麼輕量,設置有點複雜。
這裏推薦PSSH,比DSH更強大的批量管理工具,先來看下它的命令列表。

pssh 包安裝 5 個實用程序:

parallel-ssh (pssh) 在多個主機上並行地運行命令。
parallel-scp (pscp)把文件並行地複製到多個主機上。
parallel-rsync (psync)通過 rsync 協議把文件高效地並行複製到多個主機上。
parallel-slurp (pslurp)把文件並行地從多個遠程主機複製到中心主機上。
parallel-nuke (pnuke)並行地在多個遠程主機上殺死進程。

parallel-ssh [-vAiIP] [-h hosts_file] [-H [user@]host[:port]] [-l user] [-p par] [-o outdir] [-e errdir] [-t timeout] [-O options] [-x args] [-X arg] command ... 

具體參數的意義,我從官網摳下來:
-h host_file

–hosts host_file

Read hosts from the given host_file. Lines in the host file are of the
form [user@]host[:port] and can include blank lines and comments
(lines beginning with “#”). If multiple host files are given (the -h
option is used more than once), then pssh behaves as though these
files were concatenated together. If a host is specified multiple
times, then pssh will connect the given number of times.

-H

[user@]host[:port]

–host

[user@]host[:port]

-H

“[user@]host[:port] [ [user@]host[:port ] … ]”

–host

“[user@]host[:port] [ [user@]host[:port ] … ]”
Add the given host strings to the list of hosts. This option may be given multiple times, and may be used in conjunction with the -h
option.

-l user
–user user

Use the given username as the default for any host entries that don’t specifically specify a user.

-p parallelism
–par parallelism

Use the given number as the maximum number of concurrent connections.

-t timeout
–timeout timeout

Make connections time out after the given number of seconds. With a value of 0, pssh will not timeout any connections.

-o outdir
–outdir outdir

Save standard output to files in the given directory. Filenames are of
the form [user@]host[:port][.num] where the user and port are only
included for hosts that explicitly specify them. The number is a
counter that is incremented each time for hosts that are specified
more than once.

-e errdir
–errdir errdir

Save standard error to files in the given directory. Filenames are of
the same form as with the -o option.
-x args

–extra-args args

Passes extra SSH command-line arguments (see the ssh(1) man page for
more information about SSH arguments). This option may be specified
multiple times. The arguments are processed to split on whitespace,
protect text within quotes, and escape with backslashes. To pass
arguments without such processing, use the -X option instead.

-X arg
–extra-arg arg

Passes a single SSH command-line argument (see the ssh(1) man page for
more information about SSH arguments). Unlike the -x option, no
processing is performed on the argument, including word splitting. To
pass multiple command-line arguments, use the option once for each
argument.

-O options
–options options

SSH options in the format used in the SSH configuration file (see the
ssh_config(5) man page for more information). This option may be
specified multiple times.

-A
–askpass

Prompt for a password and pass it to ssh. The password may be used for
either to unlock a key or for password authentication. The password is
transferred in a fairly secure manner (e.g., it will not show up in
argument lists). However, be aware that a root user on your system
could potentially intercept the password.

-i
–inline

Display standard output and standard error as each host completes.

–inline-stdout

Display standard output (but not standard error) as each host
completes.

-v
–verbose

Include error messages from ssh with the -i and \ options.

-I
–send-input

Read input and send to each ssh process. Since ssh allows a command
script to be sent on standard input, the -I option may be used in lieu
of the command argument.

-P
–print

Display output as it arrives. This option is of limited usefulness
because output from different hosts are interleaved

終於跑起來了

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章