PHP/ beanstald:并行生成多个工人

Question

我有一个脚本，检查哪个MX记录属于一个电子邮件地址。我有大约30万封邮件要检查。所以单线程进程会花费很长时间。

我有一个队列和php是通过一个文件发送电子邮件给它。然而，我只得到一个工人执行队列。我目前无法为一个进程生成10+工人。

我运行do_job_mx.php，然后打开一个只包含电子邮件的文件，并将它们传递到队列。

从文件中获取电子邮件并放入队列的php代码- do_job_mx.php:

require_once('pheanstalk_init.php');
$pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1:11300');
$filename = '_blank.txt';
$filename = dirname(__FILE__) . '/in/' . $filename;
foreach (file($filename, FILE_SKIP_EMPTY_LINES) as $line) 
{
    $json = json_encode(array("email" => trim($line)));
    $pheanstalk
        ->useTube('process_mx')
        ->put($json);
}

worker的php代码- do_worker_process_mx.php:

class Worker 
{
    public function __construct() 
    {
        $this->log('worker process - starting');
        require_once('pheanstalk_init.php');
        $this->pheanstalk = new Pheanstalk_Pheanstalk('127.0.0.1:11300');
    }
    public function __destruct() 
    {
        $this->log('worker process - ending');
    }
    public function run() 
    {
        $this->log('worker process - starting to run');
        while(1) 
        {
            $job = $this->pheanstalk
                ->watch('process_mx')
                ->ignore('default')
                ->reserve();
            $data = json_decode($job->getData(), true);
            $this->process_mx($data);
            $this->pheanstalk->delete($job);
        }
    }
    private function process_mx($data)
    {
        $domain = explode("@", $data['email']);
        dns_get_mx($domain[1], $mx_records);
        $mx_array = explode(".", strtolower($mx_records[0]));
        $mx = array_slice($mx_array, -2, count($mx_array));
        $mx_domain = implode(".", $mx);
        echo $data['email'] . "n";
        $this->write_file($mx_domain, $data['email']);
    }       
    private function write_file($mx, $email)
    {
        $filename = fopen(dirname(__FILE__) . "/out/" . $mx . ".txt", 'ab+');
        fwrite($filename, $email . "n");
        fclose($filename);
    }
    private function log($txt) 
    {
        echo $txt . "n";
    }
}
$worker = new Worker();
$worker->run();

Supervisord相依:

[program:do_worker_process]
command=/usr/bin/php /srv/www/mydev/public_html/esp/do_worker_process_mx.php
numprocs=10
numprocs_start=10
autostart=true
autorestart=true
stopsignal=QUIT
log_stdout=true
logfile=/var/log/supervisor/worker_process_mx.log

我目前无法为一个进程生成10+工人。

运行的进程数:

# supervisorctl status
do_worker_process RUNNING    pid 44343, uptime 1:46:11

Answer 1

Centos 6自带:

beanstalkd 1.4.6主管2.1.8

我只是需要升级到supervisor 3.0。

现在我有了多个worker设施。