我有:
gcloud dataproc workflow-templates create $TEMPLATE_ID --region $REGION
gcloud beta dataproc workflow-templates set-managed-cluster $TEMPLATE_ID --region $REGION --cluster-name dailyhourlygtp --image-version 1.5
--master-machine-type=n1-standard-8 --worker-machine-type=n1-standard-16 --num-workers=10 --master-boot-disk-size=500
--worker-boot-disk-size=500 --zone=europe-west1-b
export STEP_ID=step_pyspark1
gcloud dataproc workflow-templates add-job pyspark
gs://$BUCKET_NAME/my_pyscript.py
--step-id $STEP_ID
--workflow-template $TEMPLATE_ID
--region $REGION
--jar=gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar
--initialization-actions gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar
--properties spark.jars.packages=gs://spark-lib/bigquery/spark-bigquery-latest_2.12.jar
gcloud dataproc workflow-templates instantiate $TEMPLATE_ID --region=$REGION
那么这里的问题是我如何将以下spark参数传递给我的my_pyscript.py:
--master yarn --deploy-mode cluster --conf "spark.sql.shuffle.partitions=900"
--conf "spark.sql.autoBroadcastJoinThreshold=10485760" --conf "spark.executor.memoryOverhead=8192"
--conf "spark.dynamicAllocation.enabled=true" --conf "spark.shuffle.service.enabled=true"
--executor-cores 5 --executor-memory 15g --driver-memory 16g
这在文档gcloud dataproc workflow-templates add-job pyspark:
中有描述--properties=[PROPERTY=VALUE,…]配置PySpark的键值对列表。有关可用属性的列表，请参见:https://spark.apache.org/docs/latest/configuration.html可用属性。
所以当你在没有模板的情况下向datacproc集群提交pyspark作业时，你会这样做。提交作业的spark属性通过--properties参数作为键值数组传递。
如果您的python作业需要参数，您可以在位置参数--的右侧指定它们，并以空格分隔。
对于您的示例，可以这样做:
gcloud dataproc workflow-templates add-job pyspark
gs://$BUCKET_NAME/my_pyscript.py
--step-id $STEP_ID
--workflow-template $TEMPLATE_ID
--region $REGION
--properties="spark.submit.deployMode"="cluster",
"park.sql.shuffle.partitions"="900",
"spark.sql.autoBroadcastJoinThreshold"="10485760",
"spark.executor.memoryOverhead"="8192",
"spark.dynamicAllocation.enabled"="true",
"spark.shuffle.service.enabled"="true",
"spark.executor.memory"="15g",
"spark.driver.memory"="16g",
"spark.executor.cores"="5"
-- arg1 arg2 # for named args: -- -arg1 arg1 -arg2 arg2