Flink 1.7.1 无法使用核心站点.xml对 s3a 进行身份验证

Question

使用 Flink

1.7.1 在 Kubernetes 上为单个作业集群构建它 Flink 尽管位于类路径上，但无法加载核心站点 XML，导致忽略配置，但是，如果我放置 ENV 变量AWS_SECRET_ACCESS_KEY AWS_ACCESS_KEY_ID它可以通过找到它来工作，但是如果我依赖于核心站点.xml如果没有 env 变量，它永远不会工作。

我目前正在复制核心站点.xml如 Dockerfile 所示，并且文档说HADOOP_CONF_DIR作为指向它的 env 变量。它仍然没有加载它，导致NoCredentialsProvider。

例外情况是：

Caused by: org.apache.flink.fs.s3base.shaded.com.amazonaws.AmazonClientException: No AWS Credentials provided by 
BasicAWSCredentialsProvider EnvironmentVariableCredentialsProvider InstanceProfileCredentialsProvider : org.apache.flink.fs.s3base.shaded.com.amazonaws.SdkClientException: 
Unable to load credentials from service endpoint

作业管理器/任务管理器装入的类路径

-  Classpath: 

/opt/flink-1.7.1/lib/aws-java-sdk-core-1.11.489.jar
:/opt/flink-1.7.1/lib/aws-java-sdk-kms-1.11.489.jar
:/opt/flink-1.7.1/lib/aws-java-sdk-s3-1.10.6.jar
:/opt/flink-1.7.1/lib/flink-python_2.12-1.7.1.jar
:/opt/flink-1.7.1/lib/flink-s3-fs-hadoop-1.7.1.jar
:/opt/flink-1.7.1/lib/flink-shaded-hadoop2-uber-1.7.1.jar
:/opt/flink-1.7.1/lib/hadoop-aws-2.8.0.jar:/opt/flink-1.7.1/lib/httpclient-4.5.6.jar
:/opt/flink-1.7.1/lib/httpcore-4.4.11.jar
:/opt/flink-1.7.1/lib/jackson-annotations-2.9.8.jar
:/opt/flink-1.7.1/lib/jackson-core-2.9.8.jar
:/opt/flink-1.7.1/lib/jackson-databind-2.9.8.jar
:/opt/flink-1.7.1/lib/job.jar
:/opt/flink-1.7.1/lib/joda-time-2.10.1.jar
:/opt/flink-1.7.1/lib/log4j-1.2.17.jar
:/opt/flink-1.7.1/lib/slf4j-log4j12-1.7.15.jar
:/opt/flink-1.7.1/lib/flink-dist_2.12-1.7.1.jar
:
:/hadoop/conf:

用于构建 docker 镜像的 Dockerfile：

################################################################################
#  Licensed to the Apache Software Foundation (ASF) under one
#  or more contributor license agreements.  See the NOTICE file
#  distributed with this work for additional information
#  regarding copyright ownership.  The ASF licenses this file
#  to you under the Apache License, Version 2.0 (the
#  "License"); you may not use this file except in compliance
#  with the License.  You may obtain a copy of the License at
#
#      http://www.apache.org/licenses/LICENSE-2.0
#
#  Unless required by applicable law or agreed to in writing, software
#  distributed under the License is distributed on an "AS IS" BASIS,
#  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#  See the License for the specific language governing permissions and
# limitations under the License.
################################################################################
FROM openjdk:8-jre-alpine
# Install requirements
# Modification to original Dockerfile to support rocksdb
# RUN apk add --no-cache bash snappy
# This is a fix for RocksDB compatibility

# Flink environment variables
ENV FLINK_INSTALL_PATH=/opt
ENV FLINK_HOME $FLINK_INSTALL_PATH/flink
ENV FLINK_LIB_DIR $FLINK_HOME/lib
ENV PATH $PATH:$FLINK_HOME/bin
ENV FLINK_CONF $FLINK_HOME/conf
ENV FLINK_OPT $FLINK_HOME/opt
ENV FLINK_HADOOP_CONF /hadoop/conf
# flink-dist can point to a directory or a tarball on the local system
ARG flink_dist=NOT_SET
ARG job_jar=NOT_SET
# Install build dependencies and flink
ADD $flink_dist $FLINK_INSTALL_PATH
ADD $job_jar $FLINK_INSTALL_PATH/job.jar
RUN set -x && 
  ln -s $FLINK_INSTALL_PATH/flink-* $FLINK_HOME && 
  ln -s $FLINK_INSTALL_PATH/job.jar $FLINK_LIB_DIR && 
  addgroup -S flink && adduser -D -S -H -G flink -h $FLINK_HOME flink && 
  chown -R flink:flink $FLINK_INSTALL_PATH/flink-* && 
  chown -h flink:flink $FLINK_HOME
# Modification to original Dockerfile
RUN apk add --no-cache bash libc6-compat snappy 'su-exec>=0.2'
COPY core-site.xml $FLINK_HADOOP_CONF/core-site.xml
ENV HADOOP_CONF_DIR=$FLINK_HADOOP_CONF
RUN echo "fs.hdfs.hadoopconf: $FLINK_HADOOP_CONF" >> $FLINK_CONF/flink-conf.yaml
RUN echo "akka.ask.timeout: 30 min" >> $FLINK_CONF/flink-conf.yaml
RUN echo "akka.client.timeout: 30 min" >> $FLINK_CONF/flink-conf.yaml
RUN echo "web.timeout: 180000" >> $FLINK_CONF/flink-conf.yaml
RUN mv $FLINK_OPT/flink-s3-fs-hadoop-1.7.1.jar $FLINK_LIB_DIR
COPY docker-entrypoint.sh /
RUN chmod +x docker-entrypoint.sh
RUN wget -O $FLINK_LIB_DIR/hadoop-aws-2.8.0.jar https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-aws/2.8.0/hadoop-aws-2.8.0.jar
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-s3-1.10.6.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-s3/1.10.6/aws-java-sdk-s3-1.10.6.jar
#Transitive Dependency of aws-java-sdk-s3
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-core-1.11.489.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-core/1.11.489/aws-java-sdk-core-1.11.489.jar
RUN wget -O $FLINK_LIB_DIR/aws-java-sdk-kms-1.11.489.jar http://central.maven.org/maven2/com/amazonaws/aws-java-sdk-kms/1.11.489/aws-java-sdk-kms-1.11.489.jar
RUN wget -O $FLINK_LIB_DIR/jackson-annotations-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-annotations/2.9.8/jackson-annotations-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/jackson-core-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-core/2.9.8/jackson-core-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/jackson-databind-2.9.8.jar http://central.maven.org/maven2/com/fasterxml/jackson/core/jackson-databind/2.9.8/jackson-databind-2.9.8.jar
RUN wget -O $FLINK_LIB_DIR/joda-time-2.10.1.jar http://central.maven.org/maven2/joda-time/joda-time/2.10.1/joda-time-2.10.1.jar
RUN wget -O $FLINK_LIB_DIR/httpcore-4.4.11.jar http://central.maven.org/maven2/org/apache/httpcomponents/httpcore/4.4.11/httpcore-4.4.11.jar
RUN wget -O $FLINK_LIB_DIR/httpclient-4.5.6.jar http://central.maven.org/maven2/org/apache/httpcomponents/httpclient/4.5.6/httpclient-4.5.6.jar
#Modification to original Dockerfile
USER flink
EXPOSE 8081 6123
ENTRYPOINT ["/docker-entrypoint.sh"]
CMD ["--help"]

核心站点.xml

<configuration>
    <property>
        <name>fs.s3.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
    <!-- Comma separated list of local directories used to buffer
         large results prior to transmitting them to S3. -->
    <property>
        <name>fs.s3a.buffer.dir</name>
        <value>/tmp</value>
    </property>
    <property>
        <name>fs.s3a.access.key</name>
        <description>AWS access key ID.
            Omit for IAM role-based or provider-based authentication.</description>
        <value>*</value>
    </property>
    <property>
        <name>fs.s3a.secret.key</name>
        <description>AWS secret key.
            Omit for IAM role-based or provider-based authentication.</description>
        <value>*</value>
    </property>

    <property>
        <name>fs.s3a.aws.credentials.provider</name>
        <value>org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider</value>
    </property>

</configuration>

Answer 1

好的，这已经解决了，如果你在类路径上有阴影的hadoop（将其从/opt移动到/lib），你需要在flink-conf中指定你的键，但是现在我得到以下异常

Caused by: java.io.IOException: org.apache.hadoop.fs.s3a.BasicAWSCredentialsProvider 
constructor exception.  A class specified in fs.s3a.aws.credentials.provider must provide a public constructor accepting URI and Configuration, or a public factory method named getInstance that accepts no arguments, or a public default constructor.

知道吗？

Answer 2

已解决，

将具有适当版本的hadoop-common添加到Dockerfile中有所帮助。