Spark 数据集计数需要花费大量时间

Question

>我正在使用计数函数来知道计数是否大于 0。但是，给出包含 40,00,000 个项目的特定列的大小需要 5 分钟以上。

下面是我的代码垃圾。

specficManufacturerdetailsSource = source.filter(col("ManufacturerSource").equalTo(individualManufacturerName));
specficManufacturerdetailsTarget = target.filter(col("ManufacturerTarget").equalTo(individualManufacturerName));
manufacturerSourceCount=specficManufacturerdetailsSource.count();
manufacturerTargetCount=specficManufacturerdetailsTarget.count();

System.out.println("Size of specfic manufacturer source ML :"+manufacturerSourceCount+"Size of specfic manufacturer target"+manufacturerTargetCount);
if(manufacturerSourceCount > 0 && manufacturerTargetCount > 0 ){
}

Answer 1

基于您上面提到的要求，您不需要计数。

您可以只使用 findFirst() 代替计数，如果您通过 manufacturerSourceCount.isPresent()那么这意味着count > 0