创建并行进程并等待所有进程完成，然后重做步骤

我想做的应该很简单，我自己已经找到了下面的解决方案，我所需要的只是一些指针来告诉我这是否是这样做的方法，或者我应该重构代码中的任何内容。

下面的代码，应该创建一些并行进程并等待它们完成执行，然后一次又一次地重新运行代码......

脚本在 10 分钟由 cron 作业触发一次，如果脚本正在运行，则不执行任何操作，否则启动工作进程。

任何见解都受到高度赞赏，因为我对 bash 编程并不熟悉。

#!/bin/bash
# paths
THISPATH="$( cd "$( dirname "$0" )" && pwd )"
# make sure we move in the working directory
cd $THISPATH
# console init path
CONSOLEPATH="$( cd ../../ && pwd )/console.php"
# command line arguments
daemon=0
PHPPATH="/usr/bin/php"
help=0
# flag for binary search
LOOKEDFORPHP=0
# arguments init
while getopts d:p:h: opt; do
  case $opt in
  d)
      daemon=$OPTARG
      ;;
  p)
      PHPPATH=$OPTARG
      LOOKEDFORPHP=1
      ;;
  h)
      help=$OPTARG
      ;;
  esac
done
shift $((OPTIND - 1))

# allow only one process
processesLength=$(ps aux | grep -v "grep" | grep -c $THISPATH/send-campaigns-daemon.sh)
if [ ${processesLength:-0} -gt 2 ]; then
    # The process is already running
    exit 0
fi
if [ $help -eq 1 ]; then 
    echo "---------------------------------------------------------------"
    echo "| Usage: send-campaigns-daemon.sh                             |"
    echo "| To force PHP CLI binary :                                   |"
    echo "| send-campaigns-daemon.sh -p /path/to/php-cli/binary         |"
    echo "---------------------------------------------------------------"
    exit 0
fi
# php executable path, find it if not provided
if [ $PHPPATH ] && [ ! -f $PHPPATH ] && [ $LOOKEDFORPHP -eq 0 ]; then
    phpVariants=( "php-cli" "php5-cli" "php5" "php" )
    LOOKEDFORPHP=1
    for i in "${phpVariants[@]}"
    do
        which $i >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            PHPPATH=$(which $i) 
        fi
    done
fi
if [ ! $PHPPATH ] || [ ! -f $PHPPATH ]; then
    # Did not find PHP
    exit 1
fi

# load options from app
parallelProcessesPerCampaign=3
campaignsAtOnce=10
subscribersAtOnce=300
sleepTime=30
function loadOptions {
    local COMMAND="$PHPPATH $CONSOLEPATH option get_option --name=%s --default=%d"
    parallelProcessesPerCampaign=$(printf "$COMMAND" "system.cron.send_campaigns.parallel_processes_per_campaign" 3)
    campaignsAtOnce=$(printf "$COMMAND" "system.cron.send_campaigns.campaigns_at_once" 10)
    subscribersAtOnce=$(printf "$COMMAND" "system.cron.send_campaigns.subscribers_at_once" 300)
    sleepTime=$(printf "$COMMAND" "system.cron.send_campaigns.pause" 30)
    parallelProcessesPerCampaign=$($parallelProcessesPerCampaign)
    campaignsAtOnce=$($campaignsAtOnce)
    subscribersAtOnce=$($subscribersAtOnce)
    sleepTime=$($sleepTime)
}
# define the daemon function that will stay in loop
function daemon {
    loadOptions
    local pids=() 
    local k=0 
    local i=0
    local COMMAND="$PHPPATH -q $CONSOLEPATH send-campaigns --campaigns_offset=%d --campaigns_limit=%d --subscribers_offset=%d --subscribers_limit=%d --parallel_process_number=%d --parallel_processes_count=%d --usleep=%d --from_daemon=1"
    while [ $i -lt $campaignsAtOnce ]
    do
        while [ $k -lt $parallelProcessesPerCampaign ]
        do
            parallelProcessNumber=$(( $k + 1 ))
            usleep=$(( $k * 10 + $i * 10 ))
            CMD=$(printf "$COMMAND" $i 1 $(( $subscribersAtOnce * $k )) $subscribersAtOnce $parallelProcessNumber $parallelProcessesPerCampaign $usleep)
            $CMD > /dev/null 2>&1 &
            pids+=($!)
            k=$(( k + 1 ))
        done
        i=$(( i + 1 ))
    done
    waitForPids pids
    sleep $sleepTime
    daemon
}
function daemonize {
    $THISPATH/send-campaigns-daemon.sh -d 1 -p $PHPPATH > /dev/null 2>&1 &
}
function waitForPids {
    stillRunning=0
    for i in "${pids[@]}"
    do
        if ps -p $i > /dev/null 
        then
            stillRunning=1
            break
        fi
    done
    if [ $stillRunning -eq 1 ]; then
        sleep 0.5
        waitForPids pids
    fi
    return 0
}
if [ $daemon -eq 1 ]; then
    daemon
else
    daemonize
fi
exit 0

启动脚本时，创建一个锁定文件以了解此脚本正在运行。脚本完成后，删除锁定文件。如果有人在进程运行时杀死该进程，则锁定文件将永久保留，但测试它的年龄，如果早于定义的值，则在之后删除。例如

#!/bin/bash
# 10 min
LOCK_MAX=600
typedef LOCKFILE=/var/lock/${0##*/}.lock
if [[ -f $LOCKFILE ]] ; then
    TIMEINI=$( stat -c %X $LOCKFILE )
    SEGS=$(( $(date +%s) - $TIEMPOINI ))
    if [[ $SEGS -gt $LOCK_MAX ]] ; then
        reportLocking or somethig to inform you
        # Kill old intance ???
        OLDPID=$(<$LOCKFILE)
        [[ -e /proc/$OLDPID ]] && kill -9 $OLDPID
        # Next time that the program is run, there is no lock file and it will run.
        rm $LOCKFILE
    fi
    exit 65
fi
# Save PID of this instance to the lock file
echo "$$" > $LOCKFILE
### Your code go here
# Remove the lock file before script finish
[[ -e $LOCKFILE ]] && rm $LOCKFILE
exit 0

从这里：

#!/bin/bash
...
echo PARALLEL_JOBS:${PARALLEL_JOBS:=1}
declare -a tests=($(.../find_what_to_run))
echo "${tests[@]}" | 
  xargs -d' ' -n1 -P${PARALLEL_JOBS} -I {} bash -c ".../run_that {}" || { echo "FAILURE"; exit 1; }
echo "SUCCESS"

在这里，您可以使用fuser刻痕便携式锁定的密码

好的

，所以我想我可以用经过多次测试的正确答案来回答我自己的问题。
所以这是最终版本，简化，没有注释/回声：

#!/bin/bash
sleep 2
DIR="$( cd "$( dirname "$0" )" && pwd )"
FILE_NAME="$( basename "$0" )"
COMMAND_FILE_PATH="$DIR/$FILE_NAME"
if [ ! -f "$COMMAND_FILE_PATH" ]; then 
    exit 1
fi
cd $DIR
CONSOLE_PATH="$( cd ../../ && pwd )/console.php"
PHP_PATH="/usr/bin/php"
help=0
LOOKED_FOR_PHP=0
while getopts p:h: opt; do
  case $opt in
  p)
      PHP_PATH=$OPTARG
      LOOKED_FOR_PHP=1
      ;;
  h)
      help=$OPTARG
      ;;
  esac
done
shift $((OPTIND - 1))
if [ $help -eq 1 ]; then 
    printf "%sn" "HELP INFO"
    exit 0
fi
if [ "$PHP_PATH" ] && [ ! -f "$PHP_PATH" ] && [ "$LOOKED_FOR_PHP" -eq 0 ]; then
    php_variants=( "php-cli" "php5-cli" "php5" "php" )
    LOOKED_FOR_PHP=1
    for i in "${php_variants[@]}"
    do
        which $i >/dev/null 2>&1
        if [ $? -eq 0 ]; then
            PHP_PATH="$(which $i)" 
            break
        fi
    done
fi
if [ ! "$PHP_PATH" ] || [ ! -f "$PHP_PATH" ]; then
    exit 1
fi
LOCK_BASE_PATH="$( cd ../../../common/runtime && pwd )/shell-pids"
LOCK_PATH="$LOCK_BASE_PATH/send-campaigns-daemon.pid"
function remove_lock {
    if [ -d "$LOCK_PATH" ]; then
        rmdir "$LOCK_PATH" > /dev/null 2>&1
    fi
    exit 0
}
if [ ! -d "$LOCK_BASE_PATH" ]; then
    if ! mkdir -p "$LOCK_BASE_PATH" > /dev/null 2>&1; then
        exit 1
    fi
fi
process_running=0
if mkdir "$LOCK_PATH" > /dev/null 2>&1; then
    process_running=0
else
    process_running=1
fi
if [ $process_running -eq 1 ]; then
    exit 0
fi
trap "remove_lock" 1 2 3 15
COMMAND="$PHP_PATH $CONSOLE_PATH option get_option --name=%s --default=%d"
parallel_processes_per_campaign=$(printf "$COMMAND" "system.cron.send_campaigns.parallel_processes_per_campaign" 3)
campaigns_at_once=$(printf "$COMMAND" "system.cron.send_campaigns.campaigns_at_once" 10)
subscribers_at_once=$(printf "$COMMAND" "system.cron.send_campaigns.subscribers_at_once" 300)
sleep_time=$(printf "$COMMAND" "system.cron.send_campaigns.pause" 30)
parallel_processes_per_campaign=$($parallel_processes_per_campaign)
campaigns_at_once=$($campaigns_at_once)
subscribers_at_once=$($subscribers_at_once)
sleep_time=$($sleep_time)
k=0 
i=0
pp=0
COMMAND="$PHP_PATH -q $CONSOLE_PATH send-campaigns --campaigns_offset=%d --campaigns_limit=%d --subscribers_offset=%d --subscribers_limit=%d --parallel_process_number=%d --parallel_processes_count=%d --usleep=%d --from_daemon=1"
while [ $i -lt $campaigns_at_once ]
do
    while [ $k -lt $parallel_processes_per_campaign ]
    do
        parallel_process_number=$(( $k + 1 ))
        usleep=$(( $k * 10 + $i * 10 ))
        CMD=$(printf "$COMMAND" $i 1 $(( $subscribers_at_once * $k )) $subscribers_at_once $parallel_process_number $parallel_processes_per_campaign $usleep)
        $CMD > /dev/null 2>&1 &
        k=$(( k + 1 ))
        pp=$(( pp + 1 ))
    done
    i=$(( i + 1 ))
done
wait
sleep ${sleep_time:-30}
$COMMAND_FILE_PATH -p "$PHP_PATH" > /dev/null 2>&1 &
remove_lock
exit 0

通常，它是一个锁定文件，而不是锁定路径。将 PID 保存在锁定文件中以监视进程。在这种情况下，您的锁定目录不包含任何 PID 信息。您的脚本在启动时也不会执行任何 PID 文件/目录维护，以防进程在不清理锁的情况下不正确关闭。

考虑到这一点，我更喜欢你的第一个脚本。直接监控 PID 的运行更简洁。唯一的问题是，如果使用 cron 启动第二个实例，它不知道 PID 连接到第一个实例。

您还有 processLength -gt 2，它是 2，而不是 1 个进程正在运行，因此您将复制您的进程线程。

似乎守护进程只是用守护程序调用脚本，这不是很有用。此外，拥有与函数同名的变量也是无效的。

制作锁定文件的正确方法是这样的：

# Create a temporary file
echo $$ > ${LOCKFILE}.tmp$$
# Try the lock; ln without -f is atomic 
if ln ${LOCKFILE}.tmp$$ ${LOCKFILE}; then
    # we got the lock
else
    # we didn't get the lock
fi
# Tidy up the temporary file
rm ${LOCKFILE}.tmp$$

并释放锁：

# Unlock
rm ${LOCKFILE}

关键是使用唯一名称在一侧创建锁定文件，然后尝试将其链接到真实名称。这是一个原子操作，所以它应该是安全的。

任何执行"测试和设置"的解决方案都会为您提供要处理的竞争条件。是的，这可以解决，但你最终会编写额外的代码。

相关内容

最新更新

热门标签：