BeanStalkd 做隊列服務

今天無意間看到這個倉庫講php關於 BeanStalkd 的擴展,然後就去了解了一下beanstalkd,才知道它可以用來做隊列服務。

話不多說,安裝一下試試。

首先 sudo apt search beanstalk 搜索一下發現

Sorting... Done
Full Text Search... Done
awscli/focal-updates,focal-updates 1.18.69-1ubuntu0.20.04.1 all
  Universal Command Line Environment for AWS

beanstalkd/focal,now 1.11-1 amd64 [installed]
  simple, in-memory, workqueue service

python-celery-common/focal,focal 4.2.1-5ubuntu1 all
  async task/job queue based on message passing (common files)

python-celery-doc/focal,focal 4.2.1-5ubuntu1 all
  async task/job queue based on message passing (Documentation)

python3-celery/focal,focal 4.2.1-5ubuntu1 all
  async task/job queue based on message passing (Python3 version)

ruby-beaneater/focal,focal 1.0.0-1 all
  simple beanstalkd client for Ruby

好了,找到這個beanstalkd了

然後安裝

 sudo apt-get install beanstalkd 

Reading package lists... Done
Building dependency tree       
Reading state information... Done
Suggested packages:
  doc-base
The following NEW packages will be installed:
  beanstalkd
0 upgraded, 1 newly installed, 0 to remove and 90 not upgraded.
Need to get 43.9 kB of archives.
After this operation, 125 kB of additional disk space will be used.
Get:1 http://cn.archive.ubuntu.com/ubuntu focal/universe amd64 beanstalkd amd64 1.11-1 [43.9 kB]
Get:1 http://cn.archive.ubuntu.com/ubuntu focal/universe amd64 beanstalkd amd64 1.11-1 [43.9 kB]
Fetched 31.3 kB in 4s (7,047 B/s)     
Selecting previously unselected package beanstalkd.
(Reading database ... 256731 files and directories currently installed.)
Preparing to unpack .../beanstalkd_1.11-1_amd64.deb ...
Unpacking beanstalkd (1.11-1) ...
Setting up beanstalkd (1.11-1) ...
Created symlink /etc/systemd/system/multi-user.target.wants/beanstalkd.service → /lib/systemd/system/beanstalkd.service.
beanstalkd.socket is a disabled or a static unit, not starting it.
Processing triggers for man-db (2.9.1-1) ...
Processing triggers for systemd (245.4-4ubuntu3.11) ...

安裝成功,這個東西會監聽 11300 端口

然後使用這個工具來看看

監控工具

wget https://github.com/src-d/beanstool/releases/download/v0.2.0/beanstool_v0.2.0_linux_amd64.tar.gz

獲取文件後解壓

tar -xvzf beanstool_v0.2.0_linux_amd64.tar.gz

然後拷貝到 /usr/local/bin/

sudo cp beanstool_v0.2.0_linux_amd64/beanstool /usr/local/bin/

這樣就直接用 beanstool 了

查看當前狀態

beanstool stats

結果

+---------+----------+----------+----------+----------+----------+----------+----------+
| Name    | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+---------+----------+----------+----------+----------+----------+----------+----------+
| default | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
+---------+----------+----------+----------+----------+----------+----------+----------+

然後使用composer的vendor包

composer require pda/pheanstalk

安裝完畢後創建一個input.php文件做生產者

<?php
require __DIR__ . '/vendor/autoload.php';

use Pheanstalk\Pheanstalk;

$pheanstalk = Pheanstalk::create('127.0.0.1');

// Queue a Job
$pheanstalk
  ->useTube('testtube')
  ->put("job payload goes here\n");

$pheanstalk
    ->useTube('testtube')
    ->put(
        json_encode(['test' => 'data']),  // encode data in payload
        Pheanstalk::DEFAULT_PRIORITY,     // default priority
        30, // delay by 30s
        60  // beanstalk will retry job after 60s
     );

再創建一個output.php文件做消費者

<?php
require __DIR__ . '/vendor/autoload.php';
use Pheanstalk\Pheanstalk;

$pheanstalk = Pheanstalk::create('127.0.0.1');

// we want jobs from 'testtube' only.
$pheanstalk->watch('testtube');

// this hangs until a Job is produced.
$job = $pheanstalk->reserve();

try {
    $jobPayload = $job->getData();
    // do work.

    sleep(2);
    // If it's going to take a long time, periodically
    // tell beanstalk we're alive to stop it rescheduling the job.
    $pheanstalk->touch($job);
    sleep(2);

    // eventually we're done, delete job.
    $pheanstalk->delete($job);
}
catch(\Exception $e) {
    // handle exception.
    // and let some other worker retry.
    $pheanstalk->release($job); 
}

然後執行一下  php input.php 

查看 狀態

$ beanstool stats
+----------+----------+----------+----------+----------+----------+----------+----------+
| Name     | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+----------+----------+----------+----------+----------+----------+----------+----------+
| default  | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
| testtube | 0        | 1        | 1        | 0        | 0        | 0        | 2        |
+----------+----------+----------+----------+----------+----------+----------+----------+

我們看到有一個在ready狀態,一個在delayed狀態,這是由於第二次的put採用了延時30s,然後過一段時間後再看

$ beanstool stats
+----------+----------+----------+----------+----------+----------+----------+----------+
| Name     | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+----------+----------+----------+----------+----------+----------+----------+----------+
| default  | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
| testtube | 0        | 0        | 2        | 0        | 0        | 0        | 2        |
+----------+----------+----------+----------+----------+----------+----------+----------+

已經有2個在ready狀態了。

此時我們用消費者執行一下 php output.php 

與此同時迅速看狀態

$ beanstool stats
+----------+----------+----------+----------+----------+----------+----------+----------+
| Name     | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+----------+----------+----------+----------+----------+----------+----------+----------+
| default  | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
| testtube | 0        | 0        | 1        | 1        | 0        | 0        | 2        |
+----------+----------+----------+----------+----------+----------+----------+----------+
再次執行
$ beanstool stats
+----------+----------+----------+----------+----------+----------+----------+----------+
| Name     | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+----------+----------+----------+----------+----------+----------+----------+----------+
| default  | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
| testtube | 0        | 0        | 1        | 0        | 0        | 0        | 2        |
+----------+----------+----------+----------+----------+----------+----------+----------+

因爲我們有sleep(2),所以要儘量快點操作這個狀態監控的命令,可以看到有一個拿出來放入了reserved,然後就消失了(實際上這是後面的代碼delete導致的,因爲已經消費完畢)

再次執行 php output.php 

$ beanstool stats
+----------+----------+----------+----------+----------+----------+----------+----------+
| Name     | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+----------+----------+----------+----------+----------+----------+----------+----------+
| default  | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
| testtube | 0        | 0        | 0        | 1        | 0        | 0        | 2        |
+----------+----------+----------+----------+----------+----------+----------+----------+
$ beanstool stats
+---------+----------+----------+----------+----------+----------+----------+----------+
| Name    | Buried   | Delayed  | Ready    | Reserved | Urgent   | Waiting  | Total    |
+---------+----------+----------+----------+----------+----------+----------+----------+
| default | 0        | 0        | 0        | 0        | 0        | 0        | 0        |
+---------+----------+----------+----------+----------+----------+----------+----------+

同樣也是迅速觀測這個狀態,發現消費1個,然後刪除1個,現在隊列空了,這說明確實是符合我們的期望的。

然後回到文章開頭提到的擴展,這個擴展就是幫我們實現了composer裝的那個pheanstalk包。

這個擴展如何安裝呢?

步驟如下:

克隆項目

git clone https://gitee.com/qzfzz/php-beanstalk.git

進行編譯 

phpize

然後

./configure

之後

make

再 make install

sudo make install

然後修改 php.ini

sudo gedit /etc/php/7.4/cli/php.ini /etc/php/7.4/apache2/php.ini

加上這句

extension=beanstalk

重啓apache2

sudo /etc/init.d/apache2 restart

之後就可以使用這個擴展了。

這個擴展它封裝爲函數了,可以看到他有個例子文件

簡單的找了幾個例子

<?php

$config = [
  'host' => '127.0.0.1',
  'port' => 11300
];
$beanstalk_obj = beanstalk_connect($config['host'], $config['port']);
$last_job_id = beanstalk_put($beanstalk_obj, "message detail");
beanstalk_delete($beanstalk_obj, $last_job_id);
$last_job_id = beanstalk_putInTube($beanstalk_obj, 'tubea', "message detail");

可以看到使用 connect 連接, put 塞入新的job消息, putInTube 來塞入指定管道的tubea,delete來刪除等等,具體可以看看源代碼學習一下,我對比了一下這兩種方式實現效率。

第一種採用composer包(我還特意去掉了加載文件所需要的時間)

<?phprequire __DIR__ . '/vendor/autoload.php';

use Pheanstalk\Pheanstalk;

$start = microtime( true );
$pheanstalk = Pheanstalk::create('127.0.0.1');

$pheanstalk
  ->useTube('testtube')
  ->put(date("Y-m-d H:i:s") . "job payload goes here\n");

$end = microtime(true);

echo ($end - $start) * 1000 . " ms";

執行需要2.59ms

第二種直接用擴展函數

<?php
$start = microtime(true);

$config = [
  'host' => '127.0.0.1',
  'port' => 11300
];

$beanstalk_object = beanstalk_connect($config['host'], $config['port']);
$last_job_id = beanstalk_putInTube($beanstalk_object, 'testtube', date("Y-m-d H:i:s") . "job payload goes here\n");

$end = microtime(true);

echo ($end - $start) * 1000 . " ms";

執行需要0.34ms

不得不說,擴展就是擴展,就是快的多啊!

我另外測試了一下投遞極限

循環投遞10000次消息,大概在500ms左右

$start = microtime( true );
for ($i=0; $i < 10000; $i++) { 
  $pheanstalk
  ->useTube('testtube')
  ->put(date("Y-m-d H:i:s") . "job payload goes here\n");
}
$end = microtime( true );
echo ($end - $start) * 1000 . " ms";

也就意味着1秒鐘只能投遞20000條消息,這比rabbitmq的投遞慢多了(參見這篇文章)

但是它執行1次投遞消息的時候卻比這個rabbitmq的代碼執行的快,原因是我測試了一下這個rabbitmq的連接上就耗費了9ms

$connection = new AMQPStreamConnection('localhost', 5672, 'guest', 'guest');

更別提還有這兩步了

$channel = $connection->channel();
$channel->queue_declare('hello', false, false, false, false);

這樣來看,想迅速投遞一條短消息,或者建立小信息量的job用beanstalk是很不錯的,如果有大量消息集中投遞,那使用 rabbitmq 是很不錯的。

另外這個beanstalk投遞可以延時,非常適合有些時候需要在當前時間一段時間後執行某個任務,往後弄個類似於一次性的定時器的功能,這個東西值得嘗試。

而且如果使用while死循環將 output.php 腳本一直掛着跑,它就能一直消費消息了,這樣就等於有個後端進程一直能幫我們做消費者處理堆積的任務了,特殊場景可以考慮一下這個方案。

發表評論
所有評論
還沒有人評論,想成為第一個評論的人麼? 請在上方評論欄輸入並且點擊發布.
相關文章