我正在尝试使用Dataproc(GCP(使用初始化操作。
非常简短的介绍：我们使用Spark作业在带有GCP的Dataproc集群上不断运行流程，以多种不同的方式转换我们的数据。
在过去的几天里，我正在尝试构建一个初始化操作文件(.sh(，该文件将在集群启动时运行一些Linux命令。除了最后一个命令(假设运行Spark应用程序引擎并执行作业的命令(之外，一切都很好。
所以我现在正在添加所有相关信息(我的.sh文件+错误日志(，我很想听听你的建议，同时我还在互联网上寻找答案。谢谢
我的.sh文件：
#!/bin/bash
ROLE=$(/usr/share/google/get_metadata_value attributes/dataproc-role)
if [ $ROLE = "Master" ]; then
echo $ROLE
sudo mkdir /opt/ads
sudo mkdir /opt/ads/apps
sudo gcloud compute scp --recurse 35613250742-compute@brainservice:/opt/ads/apps/ /opt/ads/apps/ --zone=europe-west2-c --internal-ip
sudo sh SparkBuildPanelBatch.sh /opt/ads/apps/apps/SparkBuildPanelBatch_Latest/DeployPack_2203.txt 20201201 20201210 20201210
fi
我的错误输出日志：
Master
WARNING: The public SSH key file for gcloud does not exist.
WARNING: The private SSH key file for gcloud does not exist.
WARNING: You do not have an SSH key for gcloud.
WARNING: SSH keygen will be executed to generate a key.
Generating public/private rsa key pair.
Your identification has been saved in /root/.ssh/google_compute_engine.
Your public key has been saved in /root/.ssh/google_compute_engine.pub.
The key fingerprint is:
SHA256:QEN6xC2DB8+lUmFPpvxilMQmPYMBXSPkaTEgNuvqP8c root@analytics-m
The key's randomart image is:
+---[RSA 2048]----+
|.oo=B@%o+ |
|..o.=/@@. |
| . *+@=. |
|. . + o |
| . o S |
|. . . |
|. . |
|. . E |
| ...o |
+----[SHA256]-----+
Updating project ssh metadata...
.............................................................................................................................................................Updated [https://www.googleapis.com/compute/v1/projects/supersal].
..done.
Waiting for SSH key to propagate.
Warning: Permanently added 'compute.4114551798115890446' (ECDSA) to the list of known hosts.
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
100 19 100 19 0 0 3022 0 --:--:-- --:--:-- --:--:-- 3166
scp: /opt/ads/apps/Profile360_Latest/serviceLog/Profile360Service.out: Permission denied
ERROR: (gcloud.compute.scp) [/usr/bin/scp] exited with return code [1].
sh: 0: Can't open SparkBuildPanelBatch.sh
我不担心这句话：'sp:/opt/ads/apps/Profile360 _Latest/serviceLog/Profile360 Service.out：权限被拒绝'因为它已停用，稍后将进行修复。我不明白为什么它不能打开.sh文件。
我想好了，我只是在最后两行之间添加了一行命令"cd/opt/ads/apps/CurrectFolder"，它很有效。哈哈。谢谢@triplee。