在Gitlab CICD中并行运行相同的重复作业

Question

有没有更好的方法来并行运行多个作业而不重复呢?

我需要同时运行多个python脚本与不同的变量，如:

python main.py --table table_1
python main.py --table table_2
python main.py --table table_3

gitlab-ci.yml看起来像这样:

extract-table1:
stage:
- run
extends:
- .execute-script
script:
- python main.py --table table_1
extract-table2:
stage:
- run
extends:
- .execute-script
script:
- python main.py --table table_2
...
and so on..

如何避免这样的重复?但是我不想循环它们因为我需要它们并行。我有10张表要做。所以重复似乎太多了。

Answer 1

您可以使用parallel:matrix如下:

extract-table:
stage:
- run
extends:
- .execute-script
script:
- python main.py --table "$TABLE_NAME" --class "$CLASS_NAME"
parallel:
matrix:
- TABLE_NAME: [table_1, table_2, table_3]
CLASS_NAME: class_1
- TABLE_NAME: [table_4, table_5, table_6]
CLASS_NAME: class_2
- TABLE_NAME: [table_7, table_8]
CLASS_NAME: [class_3, class_4]

job将运行

python main.py --table "table_1" --class "class_1"
python main.py --table "table_2" --class "class_1"
python main.py --table "table_3" --class "class_1"

python main.py --table "table_4" --class "class_2"
python main.py --table "table_5" --class "class_2"
python main.py --table "table_6" --class "class_2"

python main.py --table "table_7" --class "class_3"
python main.py --table "table_8" --class "class_3"
python main.py --table "table_7" --class "class_4"
python main.py --table "table_8" --class "class_4"