小贝子编程

在 bash 中并行执行作业



我正在尝试使用gnu-parallel并行执行某些命令 

cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomenc $x.y4m --i444 --enable-qm=1 --qm-min=$i --profile=3 -w $width -h $height -b 10 --end-usage=q --cq-level=$q -o ${x}_${q}_${i}.ivf")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("aomdec ${x}_${q}_${i}.ivf --output-bit-depth=10 -o ${x}_${q}_${i}.y4m")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("ffmpeg -nostats -loglevel 0 -y -i ${x}_${q}_${i}.y4m ${x}_${q}_${i}.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"
  cmd_arr=()
  for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
        cmd_arr+=("convert ${x}_${q}_${i}.png PNG24:${x}_${q}_${i}_ssimulacra.png")
    done
  done
  parallel --will-cite  ::: "${cmd_arr[@]}"

当我执行上面的代码时，转换似乎失败，说明它找不到要转换的 png。

执行应按该顺序进行 aomenc aomdec ffmpeg convert

它连续工作得很好

for q in 10 12 14 16 18 20 22 24 26 28 30 32 34 36 38 40; do
    for i in 4 6 8 10 12; do
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
        aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
        ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
        convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
done

这是输出，我只得到一些文件的问题，这让我相信该错误与并行有关

convert: unable to open image `bench.png_20_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_20_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_20_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_20_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_22_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_22_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_22_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_4.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_4.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_4_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_8.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_8.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_8_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_24_12.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_24_12.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_24_12_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_6.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_6.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_6_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.
convert: unable to open image `bench.png_26_10.png': No such file or directory @ error/blob.c/OpenBlob/2643.
convert: unable to open file `bench.png_26_10.png' @ error/png.c/ReadPNGImage/3733.
convert: no images defined `PNG24:bench.png_26_10_ssimulacra.png' @ error/convert.c/ConvertImageCommand/3046.

y4m 文件存在，我通过手动运行受影响文件的命令来确认它。为什么 ffmpeg 不将所有 y4m 转换为 png ？

如果您的序列号有效，那么这也应该

： 
#!/bin/bash
doit() {
    x="$1"
    q="$2"
    i="$3"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}
export -f doit
parallel doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

在使用 GNU Parallel 之前，您可以在单个组合上手动测试：

doit "$x" 10 12

通常不需要构建命令数组。

如果函数中有需要的局部（非导出）变量，则可以使用 env_parallel 而不是 parallel ：

#!/bin/bash
. `which env_parallel.bash`
# Do this in a clean environment without any variables set
# It only needs to be run once, so it does not have to be part of the script
env_parallel --record-env
# Set the local variables after doing --record-env
myvar=...
doit() {
    x="$1"
    q="$2"
    i="$3"
    echo "$myvar"
    aomenc "$x".y4m --i444 --enable-qm=1 --qm-min="$i" --profile=3 -w "$width" -h "$height" -b 10 --end-usage=q --cq-level="$q" -o "$x"_"$q"_"$i".ivf
    aomdec "$x"_"$q"_"$i".ivf --output-bit-depth=10 -o "$x"_"$q"_"$i".y4m
    ffmpeg -nostats -loglevel 0 -y -i "$x"_"$q"_"$i".y4m "$x"_"$q"_"$i".png
    convert "$x"_"$q"_"$i".png PNG24:"$x"_"$q"_"$i"_ssimulacra.png 
}
# --env _ = ignore variables defined before --record-env
env_parallel --env _ doit {1} {2} {3} ::: "$x" ::: {10..40..2} ::: {4..12..2}

考虑花一个小时浏览 https://www.gnu.org/software/parallel/parallel_tutorial.html 你的命令行会因此而爱上你。

相关内容

最新更新


热门标签：

javascript python java c# php android html jquery c++ css ios sql mysql arrays asp.net json python-3.x ruby-on-rails .net sql-server django objective-c excel regex ruby linux ajax iphone xml vba spring asp.net-mvc database wordpress string postgresql wpf windows xcode bash git oracle list vb.net multithreading eclipse algorithm macos powershell visual-studio image forms numpy scala function api selenium