- Add initialSyncCompleted flag to track CDC sequential sync completion
- Add getSyncStatus() method to CDCConsumerService
- Add /health/cdc-sync endpoint to expose sync status
- Update deploy-mining.sh to wait for CDC sync completion before calling publish APIs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
kafkacat/kcat not available in containers. Switch to kafka-console-producer
with null.marker property to send tombstone messages for Debezium offset deletion.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The Debezium Connect container uses OFFSET_STORAGE_TOPIC=debezium_offsets,
not the default connect-offsets. This fix updates the tombstone method
to use the correct topic name.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This fixes an issue where Debezium would skip initial snapshot after
full-reset because the connector offset persisted in Kafka Connect's
internal connect-offsets topic.
The fix adds two strategies to delete connector offsets:
1. REST API method (Kafka Connect 3.6+): DELETE /connectors/<name>/offsets
2. Tombstone method (Kafka Connect 3.5-): Send NULL messages via kafkacat
Reference: https://debezium.io/documentation/faq/#how_to_remove_committed_offsets_for_a_connector
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Old messages in Kafka topics were corrupting the sync because they contained
outdated data from previous resets. Now we delete the outbox topics during
full-reset to ensure clean re-sync.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The full-reset script was missing the cleanup of rwa_mining_admin.processed_events
table, which caused stale idempotency records to prevent re-consumption of
contribution outbox events after reset.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Problem: full-reset script resets Kafka consumer offsets but doesn't
clear the processed_cdc_events table. When migrations run, containers
may start and consume CDC events, inserting records into this table.
After offset reset, the service restarts and detects all events as
"duplicates" because the idempotency records still exist.
Solution: Add TRUNCATE processed_cdc_events step after CDC offset reset
in Step 8, before starting services. This ensures the idempotency table
is in sync with the Kafka offset position.
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Changes:
1. Delete Debezium connectors BEFORE dropping databases (releases replication slots)
2. Start services BEFORE registering connectors (ensures tables exist and data is synced)
3. Register connectors AFTER services sync from 1.0 CDC (snapshot.mode=initial captures existing data)
4. Add wait time for connectors to initialize before publishing data
Step order: stop services → delete connectors → drop DBs → create DBs → migrate →
start services → wait for sync → register connectors → publish data
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The full-reset function was calling contribution-records/publish-all API
which caused duplicate records in mining-admin-service because:
- Contribution records are already published to outbox when calculated
- Debezium automatically captures outbox_events and sends to Kafka
- Calling publish-all again creates duplicate events with different IDs
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Change Dockerfile to use `prisma db push` instead of `migrate deploy`
- Add publish steps for contribution records and network progress in full-reset
- This ensures new tables are created automatically when schema changes
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
hailin