提交火花程序从代码，并没有显示在历史服务器ui的工作

Question

当我从命令行提交spark作业时，作业将出现在spark历史服务器UI和hadoop资源管理器UI上，但是当我从java代码提交时，作业显示在hadoop资源管理器UI上，而不是在spark历史服务器UI上。

spark程序是一样的，只是我提交的方式不同。

提交代码:

public class submitJava {
    public static Map getPropXmlAsMap(String path) throws Exception, IOException{
        File file = new File(path);
        SAXBuilder builder = new SAXBuilder();
        Document doc = builder.build(file);
        Element root = doc.getRootElement();
        List props = root.getChildren("property");
        Map result = new HashMap();
        for(Iterator iter = props.iterator(); iter.hasNext();){
            Element element = (Element) iter.next();
            Element nameele = element.getChild("name");
            Element valueele = element.getChild("value");
            String name = nameele.getText();
            String value = valueele.getText();
            result.put(name, value);
        }
        return result;
    }
    public static void fillProperties(Configuration conf,Map map){
        Iterator iter = map.entrySet().iterator();
        while(iter.hasNext()){
            Map.Entry entry = (Map.Entry)iter.next();
            String key = (String)entry.getKey();
            String value = (String)entry.getValue();
            conf.set(key, value);
        }
    }
    public static void main(String[] args) {
        Configuration config = new Configuration();
        SparkConf sparkConf = new SparkConf();
        sparkConf.setMaster("yarn-cluster");

        try {
            fillProperties(config,getPropXmlAsMap("/home/tseg/hadoop-2.6.0/etc/hadoop/core-site.xml"));
            fillProperties(config,getPropXmlAsMap("/home/tseg/hadoop-2.6.0/etc/hadoop/yarn-site.xml"));
        } catch (Exception e) {
            e.printStackTrace();
        }
        List<String> runArgs = Arrays.asList(
                "--class","org.apache.spark.examples.JavaSparkPi",
                "--addJars","hdfs://tseg0:9010/user/tseg/hh/spark-assembly-1.4.0-hadoop2.6.0.jar",
                "--jar","file:////home/tseg/hh/spark-1.4.0-bin-hadoop2.6.0/lib/spark-examples-1.4.0-hadoop2.6.0.jar"
                ,"--arg","10"
                );
        System.setProperty("SPARK_YARN_MODE", "true");
        ClientArguments argss = new ClientArguments(runArgs.toArray(new String[runArgs.size()]),sparkConf);
        Client client = new Client(argss,config,sparkConf);
        ApplicationId applicationId = client.submitApplication();

    }

}

Answer 1

看看https://spark.apache.org/docs/1.2.0/running-on-yarn.html

它声明有一个属性spark.yarn.historyServer.address，默认为空，因为历史服务器是一个可选服务。可能在您从代码提交时没有设置，但是在您通过命令行启动时设置。