Transforming How We Run Kafka at Honeycomb
AI Summary
Honeycomb recently completed a multi-month migration of their Kafka infrastructure from self-hosted Confluent Platform and ZooKeeper on AWS EC2 to open-source Apache Kafka running in KRaft mode on AWS EKS using Strimzi. The team leveraged lessons from past incidents to design a repeatable, safe migration procedure focusing on rollback capabilities, sociotechnical process alignment, and team empowerment.
Key Takeaways
- Migrated from self-hosted ZooKeeper-based Kafka on EC2 to Apache Kafka in KRaft mode on AWS EKS using Strimzi.
- Used insights gained from previous production incidents to develop robust evacuation and migration procedures, including resetting checkpointed offsets via feature flags.
- Optimized operational load by gradually transitioning multi-team migration coordination into a streamlined process handled entirely by a single rotating team.
Why it matters
Large-scale infrastructure migrations are fraught with risk, especially when dealing with critical event streaming backbones. This article details practical strategies for turning emergency runbooks into repeatable migration procedures, managing cross-functional alignment, and distributing specialized knowledge across engineering teams.
Conversation 0
Be the first to share your perspective.