Hadoop MapReduce on Eclipse: 清理暂存区域文件:/app/hadoop/tmp/mapred

Question

2014-04-04 16:02:31.633 java[44631:1903] Unable to load realm info from SCDynamicStore
14/04/04 16:02:32 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/04/04 16:02:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
14/04/04 16:02:32 WARN mapred.JobClient: No job jar file set.  User classes may not be found. See JobConf(Class) or JobConf#setJar(String).
14/04/04 16:02:32 WARN snappy.LoadSnappy: Snappy native library not loaded
14/04/04 16:02:32 INFO mapred.FileInputFormat: Total input paths to process : 1
14/04/04 16:02:32 INFO mapred.JobClient: Cleaning up the staging area file:/app/hadoop/tmp/mapred/staging/myname183880112/.staging/job_local183880112_0001
java.lang.NullPointerException
    at org.apache.hadoop.conf.Configuration.getLocalPath(Configuration.java:950)
    at org.apache.hadoop.mapred.JobConf.getLocalPath(JobConf.java:476)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:121)
    at org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:592)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:1013)
    at org.apache.hadoop.mapred.JobClient$2.run(JobClient.java:936)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1190)
    at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:936)
    at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:910)
    at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1353)
    at LineIndex.main(LineIndex.java:92)

我正在尝试使用Eclipse中的MapReduce为Line Index执行一个Mapreduce程序。出现上述错误。我的代码是：

 public class LineIndex {
  public static class LineIndexMapper extends MapReduceBase
      implements Mapper<LongWritable, Text, Text, Text> {
    private final static Text word = new Text();
    private final static Text location = new Text();
    public void map(LongWritable key, Text val,
        OutputCollector<Text, Text> output, Reporter reporter)
        throws IOException {
      FileSplit fileSplit = (FileSplit)reporter.getInputSplit();
      String fileName = fileSplit.getPath().getName();
      location.set(fileName);
      String line = val.toString();
      StringTokenizer itr = new StringTokenizer(line.toLowerCase());
      while (itr.hasMoreTokens()) {
        word.set(itr.nextToken());
        output.collect(word, location);
      }
    }
  }

  public static class LineIndexReducer extends MapReduceBase
      implements Reducer<Text, Text, Text, Text> {
    public void reduce(Text key, Iterator<Text> values,
        OutputCollector<Text, Text> output, Reporter reporter)
        throws IOException {
      boolean first = true;
      StringBuilder toReturn = new StringBuilder();
      while (values.hasNext()){
        if (!first)
          toReturn.append(", ");
        first=false;
        toReturn.append(values.next().toString());
      }
      output.collect(key, new Text(toReturn.toString()));
    }
  }

  /**
   * The actual main() method for our program; this is the
   * "driver" for the MapReduce job.
   */
  public static void main(String[] args) {
    JobClient client = new JobClient();
    JobConf conf = new JobConf(LineIndex.class);
    conf.setJobName("LineIndexer");
    conf.setOutputKeyClass(Text.class);
    conf.setOutputValueClass(Text.class);
    FileInputFormat.addInputPath(conf, new Path("input"));
    FileOutputFormat.setOutputPath(conf, new Path("output"));

    conf.setMapperClass(LineIndexMapper.class);
    conf.setReducerClass(LineIndexReducer.class);
    conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/core-site.xml"));
    conf.addResource(new Path("/usr/local/hadoop/etc/hadoop/hdfs-site.xml"));
    client.setConf(conf);
    try {
      JobClient.runJob(conf);
    } catch (Exception e) {
      e.printStackTrace();
    }
  }
}

我无法理解和解决这里的错误空指针异常。有人可以帮我吗？

Answer 1

您可以将 mapred-site.xml 文件添加到配置类对象并重试吗？您可能还需要在该 xml 文件中指定属性mapred.local.dir。