无法在 Spark 中连接从站到主服务器



在计算引擎上使用4个实例,每个实例都使用Cloudera Manager设置。我在本地浏览器中启动master和连接没有问题,并且它连接为spark://instance-1:7077.当我在剩下的实例上启动启动从属程序时,我没有得到任何错误,直到我查看日志:

16/05/02 13:10:18 INFO worker.Worker: Started daemon with process name: 12612@instance-2.c.cluster1-1294.internal
16/05/02 13:10:18 INFO worker.Worker: Registered signal handlers for [TERM, HUP, INT]
16/05/02 13:10:18 INFO spark.SecurityManager: Changing view acls to: root
16/05/02 13:10:18 INFO spark.SecurityManager: Changing modify acls to: root
16/05/02 13:10:18 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with mod$
16/05/02 13:10:19 INFO util.Utils: Successfully started service 'sparkWorker' on port 60270.
16/05/02 13:10:19 INFO worker.Worker: Starting Spark worker 10.142.0.3:60270 with 2 cores, 6.3 GB RAM
16/05/02 13:10:19 INFO worker.Worker: Running Spark version 1.6.0
16/05/02 13:10:19 INFO worker.Worker: Spark home: /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark
16/05/02 13:10:19 ERROR worker.Worker: Failed to create work directory /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark/work

如果我使用mkdir创建"work",那么它会抛出并出错,并表示目录已经存在:

mkdir: cannot create directory ‘work’: File exists

该文件确实存在,当使用ls查找它时,它会以黑色背景的红色突出显示。如有任何帮助,我们将不胜感激。

也许这是权限问题,

试试这个,

   $sudo chown -R your_userName:your_groupName /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark

现在更改上述路径的模式

  $sudo chmod 777 /opt/cloudera/parcels/CDH-5.7.0-1.cdh5.7.0.p0.45/lib/spark

此外,所有从属设备必须具有彼此的ssh,并且能够相互交谈。

并将spark的所有配置文件也复制到从属节点。

最新更新