配置 debezium 连接器以读取没有任何部分数据的 mysql 二进制日志



我正在运行debezium从mysql bin日志中捕获数据并使用kafka-connect在Kafka上发布。我今天收到此错误:

[2019-06-18 11:32:38,175] ERROR Failed to properly convert data value for 'wallet_db_v3.user_registrations.msisdn' of type VARCHAR for row [824, 2, 2019-06-18T06:20:26]: (io.debezium.relational.TableSchemaBuilder:253)
java.lang.IllegalArgumentException: Unexpected value for JDBC type 12 and column msisdn VARCHAR(50) CHARSET utf8 NOT NULL: class=class java.lang.Integer
at io.debezium.jdbc.JdbcValueConverters.handleUnknownData(JdbcValueConverters.java:1164)
at io.debezium.jdbc.JdbcValueConverters.convertValue(JdbcValueConverters.java:1207)
at io.debezium.connector.mysql.MySqlValueConverters.convertString(MySqlValueConverters.java:380)
at io.debezium.connector.mysql.MySqlValueConverters.lambda$converter$13(MySqlValueConverters.java:293)
at io.debezium.relational.TableSchemaBuilder.lambda$createValueGenerator$2(TableSchemaBuilder.java:249)
at io.debezium.relational.TableSchema.valueFromColumnData(TableSchema.java:145)
at io.debezium.connector.mysql.RecordMakers$1.insert(RecordMakers.java:244)
at io.debezium.connector.mysql.RecordMakers$RecordsForTable.create(RecordMakers.java:456)
at io.debezium.connector.mysql.BinlogReader.handleInsert(BinlogReader.java:833)
at io.debezium.connector.mysql.BinlogReader.handleEvent(BinlogReader.java:501)
at com.github.shyiko.mysql.binlog.BinaryLogClient.notifyEventListeners(BinaryLogClient.java:1095)
at com.github.shyiko.mysql.binlog.BinaryLogClient.listenForEventPackets(BinaryLogClient.java:943)
at com.github.shyiko.mysql.binlog.BinaryLogClient.connect(BinaryLogClient.java:580)
at com.github.shyiko.mysql.binlog.BinaryLogClient$7.run(BinaryLogClient.java:825)
at java.lang.Thread.run(Thread.java:748)

表结构:

'id', 'int(10) unsigned', 'NO', 'PRI', NULL, 'auto_increment'
'msisdn', 'varchar(50)', 'NO', '', NULL, ''
'device_id', 'varchar(50)', 'NO', '', NULL, ''
'status', 'enum('waiting_for_validation','approved','rejected','expired')', 'NO', '', NULL, ''
'rejection_reason', 'enum('max_otp_validations_reached','canceled_by_user')', 'YES', '', NULL, ''
'updated_at', 'datetime', 'YES', '', 'CURRENT_TIMESTAMP', 'on update CURRENT_TIMESTAMP'
'created_at', 'datetime', 'NO', '', NULL, ''
'channel', 'varchar(50)', 'YES', '', NULL, ''

整行应如下所示:

select * from user_registrations where id = 824;
['824', '+115004937263', '56DA9A2FD4D91320D01514ED497C719E5DC9323F', 'approved', NULL, '2019-06-18 06:20:26', '2019-06-18 06:20:23', NULL]

如果现在查询该表,则此行显示为完整。

我怀疑debezium连接器在完全写入mysql二进制日志之前正在读取记录。是否有人配置debezium以确保在记录完全写入二进制日志后读取

供参考的my.cnf设置:

log-bin=/var/log/mysql/mysql-bin
binlog_format=row
binlog_row_image=full

问题出在源表 - 它是 NDB 表,不是通过设置为 full binlog_row_image参数来控制的。

相关内容

  • 没有找到相关文章

最新更新