在spark中从Mysql加载数据时出现异常

Question

我得到下面的异常，而运行spark代码从MYsql获取数据。有人能帮帮我吗?

代码在

下面

    private static final String MYSQL_CONNECTION_URL = "jdbc:mysql://localhost:3306/company";
    private static final String MYSQL_USERNAME = "test";
    private static final String MYSQL_PWD = "test123";
    private static final SparkSession sparkSession =
            SparkSession.builder().master("local[*]").appName("Spark2JdbcDs")
                       .config("spark.sql.warehouse.dir", "file:///tmp/tmp_warehouse") 
                       .getOrCreate();
    public static void main(String[] args) {
        //JDBC connection properties
        final Properties connectionProperties = new Properties();
        connectionProperties.put("user", MYSQL_USERNAME);
        connectionProperties.put("password", MYSQL_PWD);
        Dataset<Row> jdbcDF = sparkSession.sql("SELECT * FROM emp");
        List<Row> employeeFullNameRows = jdbcDF.collectAsList();
        for (Row employeeFullNameRow : employeeFullNameRows) {
            LOGGER.info(employeeFullNameRow);
        }

16/09/23 13:17:55 INFO internal。SharedState:仓库路径为file:///tmp/tmp_warehouse。16/09/23 13:17:55 INFO执行。SparkSqlParser:解析命令:SELECT * FROM emp线程"main"中的异常java.lang.UnsupportedOperationException:未被DistributedFileSystem文件系统实现org.apache.hadoop.fs.FileSystem.getScheme (FileSystem.java: 217)org.apache.hadoop.fs.FileSystem.loadFileSystems (FileSystem.java: 2624)org.apache.hadoop.fs.FileSystem.loadFileSystems (FileSystem.java: 2624)org.apache.hadoop.fs.FileSystem.getFileSystemClass (FileSystem.java: 2634)org.apache.hadoop.fs.FileSystem.createFileSystem (FileSystem.java: 2651)org.apache.hadoop.fs.FileSystem.access 200美元(FileSystem.java: 92)org.apache.hadoop.fs.FileSystem Cache.getInternal美元(FileSystem.java: 2687)org.apache.hadoop.fs.FileSystem Cache.get美元(FileSystem.java: 2669)org.apache.hadoop.fs.FileSystem.get (FileSystem.java: 371)org.apache.hadoop.fs.Path.getFileSystem (Path.java: 295)org.apache.spark.sql.catalyst.catalog.SessionCatalog.makeQualifiedPath (SessionCatalog.scala: 115)org.apache.spark.sql.catalyst.catalog.SessionCatalog.createDatabase (SessionCatalog.scala: 145)在org.apache.spark.sql.catalyst.catalog.SessionCatalog。(SessionCatalog.scala: 89)org.apache.spark.sql.internal.SessionState.catalog lzycompute美元(SessionState.scala: 95)org.apache.spark.sql.internal.SessionState.catalog (SessionState.scala: 95)在匿名org.apache.spark.sql.internal.SessionState $ $ 1美元。(SessionState.scala: 112)org.apache.spark.sql.internal.SessionState.analyzer lzycompute美元(SessionState.scala: 112)org.apache.spark.sql.internal.SessionState.analyzer (SessionState.scala: 111)在org.apache.spark.sql.execution.QueryExecution.assertAnalyzed (QueryExecution.scala: 49)org.apache.spark.sql.Dataset .ofRows美元(Dataset.scala: 64)org.apache.spark.sql.SparkSession.baseRelationToDataFrame (SparkSession.scala: 382)org.apache.spark.sql.DataFrameReader.jdbc (DataFrameReader.scala: 238)org.apache.spark.sql.DataFrameReader.jdbc (DataFrameReader.scala: 194)sparksql.sparksql1.main (sparksql1.java: 40)

下面是pom文件

    <!-- Hadoop Mapreduce Client Core -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-mapreduce-client-core</artifactId>
        <version>2.7.1</version>
    </dependency>
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-common</artifactId>
        <version>2.7.1</version>
    </dependency>
    <!-- Hadoop Core -->
    <dependency>
        <groupId>org.apache.hadoop</groupId>
        <artifactId>hadoop-core</artifactId>
        <version>1.2.1</version>
    </dependency>
    <!-- Spark  -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_2.10</artifactId>
        <version>2.0.0</version>
    </dependency>
    <!-- Spark SQL  -->
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_2.10</artifactId>
        <version>2.0.0</version>
    </dependency>
    <!-- https://mvnrepository.com/artifact/mysql/mysql-connector-java -->
    <dependency>
        <groupId>mysql</groupId>
        <artifactId>mysql-connector-java</artifactId>
        <version>5.1.20</version>
    </dependency>

Answer 1

您已经在pom.xml中添加了hadoop-core和hadoop-common。