根据文字值创建DataFrame和JavaRDD

Question

我正在用Java编写Spark应用程序，我想知道如何从文字值创建DataFrame和/或JavaRDD。

例如，我有3个整数，比如对应于字段/列(id, m_count, f_count)的(784512, 35, 40)。

Answer 1

您希望SparkContext.parallelize(...)创建JavaRDD，SQLContext.createDataFrame(...)创建数据帧。

JavaRDD rdd = sc.parallelize(Arrays.asList(1, 2, 3, 4));

如果您正在创建一个具有三个值的对象的并行列表，那么您需要：

  @Test                                                                                                 
  public void test() {                                                                   
      JavaSparkContext sc =  ...                                                    
      SQLContext sqlContext =  new SQLContext(sc);                                                      
      JavaRDD<Counter> counters = sc.parallelize(Arrays.asList(new Counter(784512, 35, 40)));           
      DataFrame countersDF = sqlContext.createDataFrame(counters, Counter.class);                       
      System.out.println(counters.collect());                                                           
      System.out.println(countersDF.collectAsList());                                                           
  }                                                                                                     

  public static class Counter implements Serializable{                                                         
      private final int id;                                                                             
      private final int m_count;                                                                        
      private final int f_count;                                                                        
      Counter(int id, int m_count, int f_count) {                                                       
          this.id = id;                                                                                 
          this.m_count = m_count;                                                                       
          this.f_count = f_count;                                                                       
      }                                                                                                 
      public String toString() {                                                                        
          return id + " " + m_count + " "  + f_count;                                                   
      }    
      // getters                                                                                          
  }