解決 多人編譯 搶佔資源問題, 使用 linux task spooler 工具.
task spooler is a Unix batch system where the tasks spooled run one after the other. The amount of
jobs to run at once can be set at any time. Each user in each system has his own job queue. The
tasks are run in the correct context (that of enqueue) from any shell/process, and its output/results
can be easily watched. It is very useful when you know that your commands depend on a lot of RAM,
a lot of disk use, give a lot of output, or for whatever reason it's better not to run them all at
the same time, while you want to keep your resources busy for maximum benfit. Its interface allows
using it easily in scripts.
task spooler 是一個簡單的批處理程序, 他可以實現簡單的隊列功能, 當你有多個任務(假設3個任務)需要執行的時候,
每個任務都是比較消耗內存, 消耗cpu資源, 消耗IO資源的任務(類似的編譯android鏡像, repo更新android代碼等等等).
如果你同時的讓這3個任務同時執行,可能每個任務執行的都很忙,每個任務都在搶佔系統資源.
這個時候你可以使用 task spooler 命令來包裝你的 3個任務(3個命令), 可以設置隊列中的執行個數是1(默認是1), 這樣
這3個任務會一次排隊的執行, 這樣的好處就是每個任務都能得到最大的系統資源, 整個下來說不定執行時間還會小於3個同時
執行的時間呢.
當一個服務器(尤其是研發的編譯服務器)多個人使用的時候,這個每個人都在搶佔系統資源,這個時候使用這個工具就能很好的
達到資源的合理利用了.每個人都能最大的使用系統資源.
默認的 tsp 工具 是每個人一個自己的隊列的, 經過我的修改, 可以實現 多人共享一個隊列.
執行很簡單的, 就是在你的命令前 加上 tsp 命令.
例如更新代碼: tsp repo sync
例如編譯: tsp make -j64
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=0/1]
$ tsp -h
usage: tsp [action] [-ngfmdE] [-L <lab>] [-D <id>] [cmd...]
Env vars:
TS_SOCKET the path to the unix socket used by the ts command.
TS_MAILTO where to mail the result (on -m). Local user by default.
TS_MAXFINISHED maximum finished jobs in the queue.
TS_MAXCONN maximum number of ts connections at once.
TS_ONFINISH binary called on job end (passes jobid, error, outfile, command).
TS_ENV command called on enqueue. Its output determines the job information.
TS_SAVELIST filename which will store the list, if the server dies.
TS_SLOTS amount of jobs which can run at once, read on server start.
TMPDIR directory where to place the output files and the default socket.
Actions:
-K kill the task spooler server
-C clear the list of finished jobs
-l show the job list (default action)
-S [num] get/set the number of max simultaneous jobs of the server.
-t [id] "tail -n 10 -f" the output of the job. Last run if not specified.
-c [id] like -t, but shows all the lines. Last run if not specified.
-p [id] show the pid of the job. Last run if not specified.
-o [id] show the output file. Of last job run, if not specified.
-i [id] show job information. Of last job run, if not specified.
-s [id] show the job state. Of the last added, if not specified.
-r [id] remove a job. The last added, if not specified.
-w [id] wait for a job. The last added, if not specified.
-k [id] send SIGTERM to the job process group. The last run, if not specified.
-u [id] put that job first. The last added, if not specified.
-U <id-id> swap two jobs in the queue.
-B in case of full queue on the server, quit (2) instead of waiting.
-h show this help
-V show the program version
Options adding jobs:
-n don't store the output of the command.
-E Keep stderr apart, in a name like the output file, but adding '.e'.
-g gzip the stored output (if not -n).
-f don't fork into background.
-m send the output by e-mail (uses sendmail).
-d the job will be run only if the job before ends well
-D <id> the job will be run only if the job of given id ends well.
-L <lab> name this task with a label, to be distinguished on listing.
-N <num> number of slots required by the job (1 default).
-O <out> output stdout and stderr to <out> file. If file already exists it will be overwritten.
使用前 就是 執行一下 tsp 命令, 如果tsp的服務端沒有運行,第一次執行tsp命令就會啓動 服務端. 服務端程序會一直運行的, 除非你執行了 tsp -K殺死它.
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=0/1]
上面就是第一次執行的效果, 輸出的隊列是空的,
執行 ps命令 可以看到 後臺有個 tsp -l 的進程了. 默認執行tsp 是 和 執行 tsp -l 效果相同的.
$ ps -ef|grep tsp
buildsr+ 35178 34925 0 17:48 pts/14 00:00:00 grep --color=auto tsp
buildsr+ 72548 1 0 14:16 ? 00:00:00 tsp -l
下面介紹幾個重要的選項.
-S 可以設置隊列同時運行的任務數. 默認是1.
如果第一次執行的命令是 tsp 加 <自己的命令> 例如下面這樣:
$ tsp ls
0
這樣ls這個命令就會放到 隊列中執行, 因爲是第一次, 這個ls命令會很快執行完成.
執行 ps命令 可以看到 後臺有個 tsp ls進程了, 這裏ls 就 你執行的哪個命令.
$ ps -ef|grep tsp
buildsr+ 35285 1 0 17:49 ? 00:00:00 tsp ls
buildsr+ 35336 34925 0 17:50 pts/14 00:00:00 grep --color=auto tsp
-c 選項 列出 某個任務的輸出, 默認最後一個任務 的輸出. -c 後面可以跟上 任務 ID 號的.
$ tsp -c
bin
jenkins
ts-out.gPuqD3
ts-out.YvWAxv
-l選項, 列出隊列的情況.
$ tsp -l
ID State Output E-Level Times(r/u/s) Command [run=0/1]
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
$ tsp -l
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
-O選項, 大寫的o, 設置output保存的文件, 默認是當前路徑下面建立個ts-out.類似的臨時文件
-D選項,設置 依賴哪個任務, 例如 -D 2, 這個任務執行 需要等待 2號 任務執行結束.
$ tsp -D 3 sleep 3
4
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3 這裏可以看到有個 && 符號
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
-S 選項, 設置隊列可以同時執行多少個任務, 默認是1個. tsp -S 2 就會設置成2個. 這樣2號等待的任務就會開始執行了.
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
$ tsp -S 2
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=2/2]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 running /home/buildsrv-ci/ts-out.pYlxed [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
設置成2 之後 就發現 有2個任務在執行了. 這個可以根據 服務器的性能 設置.
-K 關閉服務
-k 發送 SIGTERM 信號給某個運行的任務, 也就是結束這個任務的執行. 類似 kill 命令.
-C 情況隊列中的執行完畢的任務.
-U 可以調整隊列前後2個任務的順序.
-r 移除隊列中的某個任務.
-c, -t 選項 顯示任務的輸出,
其他參考
https://www.linux.com/news/queuing-tasks-batch-execution-task-spooler/