解决 多人编译 抢占资源问题, 使用 linux task spooler 工具.
task spooler is a Unix batch system where the tasks spooled run one after the other. The amount of
jobs to run at once can be set at any time. Each user in each system has his own job queue. The
tasks are run in the correct context (that of enqueue) from any shell/process, and its output/results
can be easily watched. It is very useful when you know that your commands depend on a lot of RAM,
a lot of disk use, give a lot of output, or for whatever reason it's better not to run them all at
the same time, while you want to keep your resources busy for maximum benfit. Its interface allows
using it easily in scripts.
task spooler 是一个简单的批处理程序, 他可以实现简单的队列功能, 当你有多个任务(假设3个任务)需要执行的时候,
每个任务都是比较消耗内存, 消耗cpu资源, 消耗IO资源的任务(类似的编译android镜像, repo更新android代码等等等).
如果你同时的让这3个任务同时执行,可能每个任务执行的都很忙,每个任务都在抢占系统资源.
这个时候你可以使用 task spooler 命令来包装你的 3个任务(3个命令), 可以设置队列中的执行个数是1(默认是1), 这样
这3个任务会一次排队的执行, 这样的好处就是每个任务都能得到最大的系统资源, 整个下来说不定执行时间还会小于3个同时
执行的时间呢.
当一个服务器(尤其是研发的编译服务器)多个人使用的时候,这个每个人都在抢占系统资源,这个时候使用这个工具就能很好的
达到资源的合理利用了.每个人都能最大的使用系统资源.
默认的 tsp 工具 是每个人一个自己的队列的, 经过我的修改, 可以实现 多人共享一个队列.
执行很简单的, 就是在你的命令前 加上 tsp 命令.
例如更新代码: tsp repo sync
例如编译: tsp make -j64
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=0/1]
$ tsp -h
usage: tsp [action] [-ngfmdE] [-L <lab>] [-D <id>] [cmd...]
Env vars:
TS_SOCKET the path to the unix socket used by the ts command.
TS_MAILTO where to mail the result (on -m). Local user by default.
TS_MAXFINISHED maximum finished jobs in the queue.
TS_MAXCONN maximum number of ts connections at once.
TS_ONFINISH binary called on job end (passes jobid, error, outfile, command).
TS_ENV command called on enqueue. Its output determines the job information.
TS_SAVELIST filename which will store the list, if the server dies.
TS_SLOTS amount of jobs which can run at once, read on server start.
TMPDIR directory where to place the output files and the default socket.
Actions:
-K kill the task spooler server
-C clear the list of finished jobs
-l show the job list (default action)
-S [num] get/set the number of max simultaneous jobs of the server.
-t [id] "tail -n 10 -f" the output of the job. Last run if not specified.
-c [id] like -t, but shows all the lines. Last run if not specified.
-p [id] show the pid of the job. Last run if not specified.
-o [id] show the output file. Of last job run, if not specified.
-i [id] show job information. Of last job run, if not specified.
-s [id] show the job state. Of the last added, if not specified.
-r [id] remove a job. The last added, if not specified.
-w [id] wait for a job. The last added, if not specified.
-k [id] send SIGTERM to the job process group. The last run, if not specified.
-u [id] put that job first. The last added, if not specified.
-U <id-id> swap two jobs in the queue.
-B in case of full queue on the server, quit (2) instead of waiting.
-h show this help
-V show the program version
Options adding jobs:
-n don't store the output of the command.
-E Keep stderr apart, in a name like the output file, but adding '.e'.
-g gzip the stored output (if not -n).
-f don't fork into background.
-m send the output by e-mail (uses sendmail).
-d the job will be run only if the job before ends well
-D <id> the job will be run only if the job of given id ends well.
-L <lab> name this task with a label, to be distinguished on listing.
-N <num> number of slots required by the job (1 default).
-O <out> output stdout and stderr to <out> file. If file already exists it will be overwritten.
使用前 就是 执行一下 tsp 命令, 如果tsp的服务端没有运行,第一次执行tsp命令就会启动 服务端. 服务端程序会一直运行的, 除非你执行了 tsp -K杀死它.
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=0/1]
上面就是第一次执行的效果, 输出的队列是空的,
执行 ps命令 可以看到 后台有个 tsp -l 的进程了. 默认执行tsp 是 和 执行 tsp -l 效果相同的.
$ ps -ef|grep tsp
buildsr+ 35178 34925 0 17:48 pts/14 00:00:00 grep --color=auto tsp
buildsr+ 72548 1 0 14:16 ? 00:00:00 tsp -l
下面介绍几个重要的选项.
-S 可以设置队列同时运行的任务数. 默认是1.
如果第一次执行的命令是 tsp 加 <自己的命令> 例如下面这样:
$ tsp ls
0
这样ls这个命令就会放到 队列中执行, 因为是第一次, 这个ls命令会很快执行完成.
执行 ps命令 可以看到 后台有个 tsp ls进程了, 这里ls 就 你执行的哪个命令.
$ ps -ef|grep tsp
buildsr+ 35285 1 0 17:49 ? 00:00:00 tsp ls
buildsr+ 35336 34925 0 17:50 pts/14 00:00:00 grep --color=auto tsp
-c 选项 列出 某个任务的输出, 默认最后一个任务 的输出. -c 后面可以跟上 任务 ID 号的.
$ tsp -c
bin
jenkins
ts-out.gPuqD3
ts-out.YvWAxv
-l选项, 列出队列的情况.
$ tsp -l
ID State Output E-Level Times(r/u/s) Command [run=0/1]
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
$ tsp -l
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
-O选项, 大写的o, 设置output保存的文件, 默认是当前路径下面建立个ts-out.类似的临时文件
-D选项,设置 依赖哪个任务, 例如 -D 2, 这个任务执行 需要等待 2号 任务执行结束.
$ tsp -D 3 sleep 3
4
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3 这里可以看到有个 && 符号
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
-S 选项, 设置队列可以同时执行多少个任务, 默认是1个. tsp -S 2 就会设置成2个. 这样2号等待的任务就会开始执行了.
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=1/1]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 queued (file) [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
$ tsp -S 2
$ tsp
ID State Output E-Level Times(r/u/s) Command [run=2/2]
1 running /home/buildsrv-ci/ts-out.zDDm5V [buildsrv-ci]sleep 500
2 running /home/buildsrv-ci/ts-out.pYlxed [buildsrv-ci]sleep 100
3 queued (file) [buildsrv-ci]sleep 200
4 queued (file) [3]&& [buildsrv-ci]sleep 3
0 finished /home/buildsrv-ci/ts-out.gPuqD3 0 0.00/0.00/0.00 [buildsrv-ci]ls
设置成2 之后 就发现 有2个任务在执行了. 这个可以根据 服务器的性能 设置.
-K 关闭服务
-k 发送 SIGTERM 信号给某个运行的任务, 也就是结束这个任务的执行. 类似 kill 命令.
-C 情况队列中的执行完毕的任务.
-U 可以调整队列前后2个任务的顺序.
-r 移除队列中的某个任务.
-c, -t 选项 显示任务的输出,
其他参考
https://www.linux.com/news/queuing-tasks-batch-execution-task-spooler/