Kafka 弹簧启动应用程序生产者，并且无法使用 Avro 格式的 Kafka Sink 连接器来反映这一点

Question

我的目标是我有具有 Avro 序列化属性的Spring 启动应用程序 kafka 生产者，我除了推送到相应主题的消息之外，该主题应通过融合接收器连接器访问并插入到 mysql/Oracle 数据库表中，我能够生成 Avro 序列化和Spring 启动消费者可以Avro反序列化但我的接收器连接器不起作用， 我无法捕捉到接收器连接器是哪种有效负载，以及应该如何编码 spring 启动生产者以推送消息，以便接收器连接器可以应对该属性

提前致谢:)

这是 application.yml in spring boot application

server: 
port: 9000
spring.kafka:
bootstrap-servers: "localhost:9092"
properties:
schema.registry.url: "http://localhost:8081"
specific.avro.reader: true
producer:
key-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
value-serializer: "io.confluent.kafka.serializers.KafkaAvroSerializer"
app:
topic: event_pay2

这是从 spring 引导应用程序创建模式的有效负载

{
"schema": {
"type": "struct",
"fields": [
{
"type": "string",
"optional": false,
"field": "userid"
},
{
"type": "string",
"optional": false,
"field": "regionid"
},
{
"type": "string",
"optional": false,
"field": "gender"
}
],
"optional": false,
"name": "oracle.user"
},
"payload": {
"userid": "User_1",
"regionid": "Region_5",
"gender": "FEMALE"
}
}

庞.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.0.5.RELEASE</version>
<relativePath/> <!-- lookup parent from repository -->
</parent>
<groupId>com.kafka</groupId>
<artifactId>kafka-producer-example</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>kafka-producer-example</name>
<description>Demo project for Spring Boot</description>
<repositories>
<repository>
<id>confluent</id>
<url>http://packages.confluent.io/maven</url>
</repository>
</repositories>
<properties>
<java.version>1.8</java.version>
<confluent.version>4.0.0</confluent.version>
</properties>
<dependencies>
<dependency>
<groupId>com.google.code.gson</groupId>
<artifactId>gson</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-devtools</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.kafka</groupId>
<artifactId>spring-kafka-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
<version>${confluent.version}</version>
<exclusions>
<exclusion>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-log4j12</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.avro</groupId>
<artifactId>avro</artifactId>
<version>1.8.2</version>
</dependency>
</dependencies>
<build>
<pluginManagement>
<plugins>
<plugin>
<groupId>org.apache.avro</groupId>
<artifactId>avro-maven-plugin</artifactId>
<version>1.8.2</version>
<executions>
<execution>
<phase>generate-sources</phase>
<goals>
<goal>schema</goal>
</goals>
<configuration>
<sourceDirectory>${project.basedir}/src/main/resources/avro/</sourceDirectory>
<outputDirectory>${project.build.directory}/generated/avro</outputDirectory>
</configuration>
</execution>
</executions>
</plugin>

<plugin>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-maven-plugin</artifactId>
</plugin>
</plugins>
</pluginManagement>
</build>
</project>

这是我的休息电话如何将消息推送到卡夫卡主题

@PostMapping("/publish/avrodata")
public String sendMessage(@RequestBody String request) {
sender.send(request);
return "Published successfully";
}

最后我的水槽连接器

"name": "JDBC_Sink_EVENT_PAY",
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": "1",
"topics": "event_pay2",
"connection.url": "jdbc:mysql://localhost:3306/user",
"connection.user": "****",
"connection.password": "****",
"auto.create": "true",
"auto.evolve":"true",
"key.converter": "io.confluent.connect.avro.AvroConverter",
"key.converter.schema.registry.url": "http://localhost:8081",
"key.converter.schemas.enable": "true",
"value.converter": "io.confluent.connect.avro.AvroConverter",
"value.converter.schema.registry.url": "http://localhost:8081",
"value.converter.schemas.enable": "true"

Answer 1

始终，始终在设置连接器之前调试主题。为此使用kafka-avro-console-consumer。如果这不起作用，那么Connect +AvroConverter可能也不起作用(根据我的经验)，您可以减少问题空间。

---

如果我正确阅读了您的代码，那么您发送的是一个字符串，而不是一个 Avro 对象

public String sendMessage(@RequestBody String request) {
sender.send(request);  // <--- here and ^^^^^^ here
return "Published successfully";
}

相反，您需要将输入请求解析为作为/src/main/resources/avro架构数据的一部分创建的对象，而不仅仅是作为字符串通过传入请求进行转发。

那个AVSC文件可能看起来像

{
"type": "Record",
"namespace": "oracle.user",
"name": "User",
"fields": [
{ "type": "string", "name: "user_id" },
{ "type": "string", "name": "regionid" },
{ "type" "string", "name" "gender" }
]
}

这将创建一个oracle.user.User对象，使您的KafkaTemplate需要类似于KafkaTemplate<String, oracle.user.User> sender