fix: RegexRouter 精确匹配 outbox 表,防止 heartbeat 事件污染消费者
问题根因 (rwadurian 生产环境发现):
- transforms.route.regex=".*" 将所有 CDC 变更事件(包括 heartbeat 表)
统一路由到 outbox topic,导致消费者收到非 outbox 格式的消息
- 消费者无法解析 heartbeat 消息 → prisma:error + "Unknown event format" 告警
- 每 10 秒一次 heartbeat × 5 个 connector = 持续的日志噪音
修复方案:
- transforms.route.regex 从 ".*" 改为 ".*outbox"
- 只有匹配 *outbox 的 topic(即真正的 outbox 表变更)才被路由到目标 topic
- heartbeat/signal 表的变更走默认 topic 命名(无消费者监听,自动过期)
- heartbeat 仍在 table.include.list 中,确保 Kafka offset 正常推进
→ confirmed_flush_lsn 持续前进 → WAL 不积压
验证 (rwadurian 生产环境):
- 修复后 mining-admin-service "Unknown event format" 告警清零
- WAL lag 从 540MB 降至 30-97MB 正常波动
- 5 个 connector 全部 RUNNING,heartbeat 正常(5s 内更新)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
127f2cdd81
commit
9e0152ca0d
|
|
@ -31,7 +31,7 @@
|
|||
"transforms.unwrap.delete.handling.mode": "rewrite",
|
||||
"transforms.unwrap.add.fields": "op,table,source.ts_ms",
|
||||
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
|
||||
"transforms.route.regex": ".*",
|
||||
"transforms.route.regex": ".*outbox",
|
||||
"transforms.route.replacement": "cdc.genex.outbox",
|
||||
|
||||
"heartbeat.interval.ms": "10000",
|
||||
|
|
|
|||
Loading…
Reference in New Issue