fix: RegexRouter 精确匹配 outbox 表,防止 heartbeat 事件污染消费者

问题根因 (rwadurian 生产环境发现):
  - transforms.route.regex=".*" 将所有 CDC 变更事件(包括 heartbeat 表)
    统一路由到 outbox topic,导致消费者收到非 outbox 格式的消息
  - 消费者无法解析 heartbeat 消息 → prisma:error + "Unknown event format" 告警
  - 每 10 秒一次 heartbeat × 5 个 connector = 持续的日志噪音

修复方案:
  - transforms.route.regex 从 ".*" 改为 ".*outbox"
  - 只有匹配 *outbox 的 topic(即真正的 outbox 表变更)才被路由到目标 topic
  - heartbeat/signal 表的变更走默认 topic 命名(无消费者监听,自动过期)
  - heartbeat 仍在 table.include.list 中,确保 Kafka offset 正常推进
    → confirmed_flush_lsn 持续前进 → WAL 不积压

验证 (rwadurian 生产环境):
  - 修复后 mining-admin-service "Unknown event format" 告警清零
  - WAL lag 从 540MB 降至 30-97MB 正常波动
  - 5 个 connector 全部 RUNNING,heartbeat 正常(5s 内更新)

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
hailin 2026-02-15 07:04:17 -08:00
parent 127f2cdd81
commit 9e0152ca0d
1 changed files with 1 additions and 1 deletions

View File

@ -31,7 +31,7 @@
"transforms.unwrap.delete.handling.mode": "rewrite",
"transforms.unwrap.add.fields": "op,table,source.ts_ms",
"transforms.route.type": "org.apache.kafka.connect.transforms.RegexRouter",
"transforms.route.regex": ".*",
"transforms.route.regex": ".*outbox",
"transforms.route.replacement": "cdc.genex.outbox",
"heartbeat.interval.ms": "10000",