pyspark代码由于端口问题而失败

Question

我有一个小的pyspark代码，它在我的本地机器上写入csv文件。每次我运行代码时，它都使用不同的端口，因为以前的端口无法绑定。这是错误代码。我如何使用相同的端口一遍又一遍，而运行相同的代码多次

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4046. Attempting port 4047.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4047. Attempting port 4048.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4048. Attempting port 4049.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4049. Attempting port 4050.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4050. Attempting port 4051.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4051. Attempting port 4052.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4052. Attempting port 4053.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4053. Attempting port 4054.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4054. Attempting port 4055.
21/08/31 21:05:45 WARN Utils: Service 'SparkUI' could not bind on port 4055. Attempting port 4056.
21/08/31 21:05:45 ERROR SparkUI: Failed to bind SparkUI
java.net.BindException: Failed to bind to /0.0.0.0:4056: Service 'SparkUI' failed after 16 retries (starting from 4040)! Consider explicitly setting the appropriate port for the service 'SparkUI' (for example spark.ui.port for SparkUI) to an available port or increasing spark.port.maxRetries.
at org.sparkproject.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:346)
at org.sparkproject.jetty.server.ServerConnector.open(ServerConnector.java:308)
at org.sparkproject.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:80)
at org.sparkproject.jetty.server.ServerConnector.doStart(ServerConnector.java:236)
at org.sparkproject.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:68)
at org.apache.spark.ui.JettyUtils$.newConnector$1(JettyUtils.scala:301)
at org.apache.spark.ui.JettyUtils$.httpConnect$1(JettyUtils.scala:332)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5(JettyUtils.scala:335)
at org.apache.spark.ui.JettyUtils$.$anonfun$startJettyServer$5$adapted(JettyUtils.scala:335)
at org.apache.spark.util.Utils$.$anonfun$startServiceOnPort$2(Utils.scala:2256)
at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158)
at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:2248)
at org.apache.spark.ui.JettyUtils$.startJettyServer(JettyUtils.scala:336)
at org.apache.spark.ui.WebUI.bind(WebUI.scala:146)
at org.apache.spark.SparkContext.$anonfun$new$10(SparkContext.scala:470)
at org.apache.spark.SparkContext.$anonfun$new$10$adapted(SparkContext.scala:470)
at scala.Option.foreach(Option.scala:407)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:470)
at org.apache.spark.api.java.JavaSparkContext.<init>(JavaSparkContext.scala:58)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:247)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357)
at py4j.Gateway.invoke(Gateway.java:238)
at py4j.commands.ConstructorCommand.invokeConstructor(ConstructorCommand.java:80)
at py4j.commands.ConstructorCommand.execute(ConstructorCommand.java:69)
at py4j.GatewayConnection.run(GatewayConnection.java:238)
at java.lang.Thread.run(Thread.java:748)

Answer 1

看起来您有Spark作业正在运行，但没有正确退出。正确退出这些应用程序，然后您将能够一直使用端口4040。