Tech Stack -
Spark - 2.4.7,
Scala - 2.11.8,
Running On AWS EMR
所以我试图将kineesis流写入特定的s3位置，但对于一些s3问题，我无法这样做。另一个有趣的观察是，在Databricks上，相同的代码工作得很好。
My Code -
transformations.materialization()
.writeStream
.trigger(Trigger.ProcessingTime(triggerMatInterval))
.outputMode(OutputMode.Append())
.format(DataFormat)
.option("path", Path)
.option("checkpointLocation", CheckpointPath) \Error Takes place here
.start()
S3 Location格式- s3a://abcd/ef/in/gh/checkpoints/ijk
错误——
Exception in thread "main" java.nio.file.AccessDeniedException: s3a://<S3Location>: innerMkdirs on <Same S3 Location>: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 77EA9D8F5F71C60C; S3 Extended Request ID: EH3Su6H/hcQjW/5E6+vEPjWyyy62OKb+CgoU1SCJKzMTt41IfVm8sJLDJTOYo8iIZwrN7GvD4wU=; Proxy: null), S3 Extended Request ID: EH3Su6H/hcQjW/5E6+vEPjWyyy62OKb+CgoU1SCJKzMTt41IfVm8sJLDJTOYo8iIZwrN7GvD4wU=
...
...
...
...
...
...
...
...
...
Caused by: com.amazonaws.services.s3.model.AmazonS3Exception: Access Denied (Service: Amazon S3; Status Code: 403; Error Code: AccessDenied; Request ID: 77EA9D8F5F71C60C; S3 Extended Request ID: EH3Su6H/hcQjW/5E6+vEPjWyyy62OKb+CgoU1SCJKzMTt41IfVm8sJLDJTOYo8iIZwrN7GvD4wU=; Proxy: null), S3 Extended Request ID: EH3Su6H/hcQjW/5E6+vEPjWyyy62OKb+CgoU1SCJKzMTt41IfVm8sJLDJTOYo8iIZwrN7GvD4wU=
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1828)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleServiceErrorResponse(AmazonHttpClient.java:1412)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1374)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1145)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
at com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
at com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5219)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:5165)
at com.amazonaws.services.s3.AmazonS3Client.access$300(AmazonS3Client.java:405)
at com.amazonaws.services.s3.AmazonS3Client$PutObjectStrategy.invokeServiceCall(AmazonS3Client.java:6180)
at com.amazonaws.services.s3.AmazonS3Client.uploadObject(AmazonS3Client.java:1824)
at com.amazonaws.services.s3.AmazonS3Client.putObject(AmazonS3Client.java:1784)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.uploadInOneChunk(UploadCallable.java:168)
at com.amazonaws.services.s3.transfer.internal.UploadCallable.call(UploadCallable.java:148)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:115)
at com.amazonaws.services.s3.transfer.internal.UploadMonitor.call(UploadMonitor.java:45)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Command exiting with ret '1'
Thanks In Advance.
- 尝试改变s3a://<>在s3://& lt;在
- 确保您可以从EMR所在的相同帐户访问桶
无论您以谁的身份登录，都没有对桶的写访问权限，或者至少没有试图写到
的路径