Flink,join DataSet数据集，join实例，java版本

电玩女神 2021-09-24 10:50 329阅读 0赞

package join;
 
 import org.apache.flink.api.common.functions.JoinFunction;
 import org.apache.flink.api.java.DataSet;
 import org.apache.flink.api.java.ExecutionEnvironment;
 import org.apache.flink.api.java.tuple.Tuple2;
 
 /**
 * @Author you guess
 * @Date 2020/6/17 09:45
 * @Version 1.0
 * @Desc
 */
 public class JoinDataSetTest {
 
 public static void main(String[] args) throws Exception {
 ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment();
 
 DataSet<Tuple2<Long, String>> source1 = env.fromElements(
 Tuple2.of(1L, "xiaoming"),
 Tuple2.of(2L, "xiaowang"));
 
 DataSet<Tuple2<Long, String>> source2 = env.fromElements(
 Tuple2.of(2L, "xiaoli"),
 Tuple2.of(2L, "xiaohong"),
 Tuple2.of(1L, "shinelon"),
 Tuple2.of(1L, "zhangsan"),
 Tuple2.of(3L, "hhhhhh"));
 
 source1.join(source2)
 .where(0).equalTo(0)
 .with(new JoinFunction<Tuple2<Long, String>, Tuple2<Long, String>, Object>() {
 @Override
 public Object join(Tuple2<Long, String> first, Tuple2<Long, String> second) throws Exception {
 return Tuple2.of(first.f0, first.f1 + "," + second.f1);
 }
 }).print();
 
 //处理数据集不用显式调用execute，不然执行抛异常java.lang.RuntimeException。处理数据流才需要调用execute
 //env.execute("Flink CoGroupDataSetTest by Java ");
 }
 }

输出：笛卡尔积运算。符合inner join 。'3L'匹配不到，所以不输出。

(1,xiaoming,shinelon)  
(1,xiaoming,zhangsan)  
(2,xiaowang,xiaoli)  
(2,xiaowang,xiaohong)

使用FlatJoinFunction:

source1.join(source2)
 .where(0).equalTo(0)
 .with(new FlatJoinFunction<Tuple2<Long, String>, Tuple2<Long, String>, Object>() {
 @Override
 public void join(Tuple2<Long, String> first, Tuple2<Long, String> second, Collector<Object> out) throws Exception {
 out.collect(Tuple2.of(first.f0, first.f1 + "," + second.f1));
 }
 }).print();

输出同上。

flink版本1.9.2,java版本1.8

源码：

/**
 * Interface for Join functions. Joins combine two data sets by joining their
 * elements on specified keys. This function is called with each pair of joining elements.
 *
 * By default, the joins follows strictly the semantics of an "inner join" in SQL.
 * the semantics are those of an "inner join", meaning that elements are filtered out
 * if their key is not contained in the other data set.
 *
 * The basic syntax for using Join on two data sets is as follows:
 * <pre>{@code
 * DataSet<X> set1 = ...;
 * DataSet<Y> set2 = ...;
 *
 * set1.join(set2).where(<key-definition>).equalTo(<key-definition>).with(new MyJoinFunction());
 * }</pre>
 *
 * {@code set1} is here considered the first input, {@code set2} the second input.
 *
 * The Join function is an optional part of a join operation. If no JoinFunction is provided,
 * the result of the operation is a sequence of 2-tuples, where the elements in the tuple are those that
 * the JoinFunction would have been invoked with.
 *
 * Note: You can use a {@link CoGroupFunction} to perform an outer join.
 *
 * @param <IN1> The type of the elements in the first input.
 * @param <IN2> The type of the elements in the second input.
 * @param <OUT> The type of the result elements.
 */
 @Public
 @FunctionalInterface
 public interface JoinFunction<IN1, IN2, OUT> extends Function, Serializable {
 
 	/**
 	 * The join method, called once per joined pair of elements.
 	 *
 	 * @param first The element from first input.
 	 * @param second The element from second input.
 	 * @return The resulting element.
 	 *
 	 * @throws Exception This method may throw exceptions. Throwing an exception will cause the operation
 	 * to fail and may trigger recovery.
 	 */
 	OUT join(IN1 first, IN2 second) throws Exception;
 }