如何访问组合 KV 对时组合 CombinerFn 子类中的密钥

Question

我正在使用CombineFn的子类来实现CombinePerKeyExample，而不是使用SerializableFunction的实现

package me.examples;
import org.apache.beam.sdk.coders.AvroCoder;
import org.apache.beam.sdk.coders.DefaultCoder;
import org.apache.beam.sdk.transforms.Combine.CombineFn;
import java.util.HashSet;
import java.util.Set;
public class ConcatWordsCombineFn extends CombineFn<String, ConcatWordsCombineFn.Accumulator, String> {
    @DefaultCoder(AvroCoder.class)
    public static class Accumulator{
        HashSet<String> plays;
    }
    @Override
    public Accumulator createAccumulator(){
        Accumulator accumulator = new Accumulator();
        accumulator.plays = new HashSet<>();
        return accumulator;
    }
    @Override
    public Accumulator addInput(Accumulator accumulator, String input){
        accumulator.plays.add(input);
        return accumulator;
    }
    @Override
    public Accumulator mergeAccumulators(Iterable<Accumulator> accumulators){
        Accumulator mergeAccumulator = new Accumulator();
        mergeAccumulator.plays = new HashSet<>();
        for(Accumulator accumulator: accumulators){
            mergeAccumulator.plays.addAll(accumulator.plays);
        }
        return mergeAccumulator;
    }
    @Override
    public String extractOutput(Accumulator accumulator){
        //how to access the key here ? 
        return String.join(",", accumulator.plays);
    }
}

管道由ReadFromBigQuery、ExtractAllPlaysOfWords(下面的代码(和WriteToBigQuery

package me.examples;
import com.google.api.services.bigquery.model.TableRow;
import org.apache.beam.sdk.coders.KvCoder;
import org.apache.beam.sdk.coders.StringUtf8Coder;
import org.apache.beam.sdk.transforms.Combine;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.transforms.ParDo;
import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PCollection;
public class PlaysForWord extends PTransform<PCollection<TableRow>, PCollection<TableRow>> {

    @Override
    public PCollection<TableRow> expand(PCollection<TableRow> input) {
            PCollection<KV<String, String>> largeWords = input.apply("ExtractLargeWords", ParDo.of(new ExtractLargeWordsFn()));
            PCollection<KV<String, String>> wordNPlays = largeWords.apply("CombinePlays",Combine.perKey(new ConcatWordsCombineFn()));
            wordNPlays.setCoder(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()));
            PCollection<TableRow> rows = wordNPlays.apply("FormatToRow", ParDo.of(new FormatShakespeareOutputFn()));
            return rows;
    }
}

我想在ConcatWordsCombineFn访问密钥，以便在此基础上进行最终积累。例如，如果键以a开头，则可以将单词与,连接起来，或者;否则使用。

查看编程指南时

如果需要根据键更改组合策略(例如，某些用户的 MIN，其他用户的 MAX(，则可以定义 KeyedCombineFn 以访问组合策略中的键。

我在org.apache.beam.sdk.transforms.Combine找不到KeyedCombineFn我正在使用Apache Beam 2.12.0和Google Dataflow作为运行器。

Answer 1

我认为没有内置的方法可以解决这个问题。简单的解决方法(我知道并不完美(是将您的字符串包装到另一个 KV：KV<String, KV<String, String>> 两个键相同。