Kafka Pipeline Architect
Design production Kafka pipelines with verified performance and scale
What You Can Do
You can architect production-ready Kafka pipelines with schema validation, partition strategies, and lag management. Claude helps you identify bottlenecks, optimize throughput, and design fault-tolerant systems with concrete performance guarantees. You'll get architecture diagrams, configuration templates, and troubleshooting playbooks for real-world streaming challenges.
Features
Balance load across brokers based on key cardinality and throughput
Validate message format, handle schema evolution, prevent data corruption
Design metrics dashboards and alert thresholds for consumer lag
Design replication, exactly-once semantics, and crash recovery workflows
Identify bottlenecks in producers, brokers, and consumers with concrete tuning parameters
Calculate broker disk, memory, network requirements from message volume projections
Design cross-region replication, disaster recovery, and migration strategies
Example Output
Example 1: Partition Strategy Analysis
Topic: user-events
Current partitions: 4
Recommended partitions: 12
Rationale: Key cardinality ~2M users, target throughput 50K msgs/sec
Partition assignment: Round-robin on user_id hash
Expected rebalance time: 2.3s
Example 2: Consumer Lag Diagnosis
Consumer group: analytics-processor
Lag by partition:
- P0: 1.2M messages (12 mins behind)
- P1: 450K messages (4.5 mins behind)
- P2: 50K messages (30 secs behind)
Root cause: Consumer thread on P0 processing at 1.6K msgs/sec vs 2.1K msgs/sec ingest
Recommendation: Increase consumer parallelism or add 2 more consumer instances
Example 3: Schema Evolution Checklist
✓ New field: user_country (string, optional)
✓ Default value: 'UNKNOWN'
✓ Backward compatibility: Old producers can read new schema
✓ Forward compatibility: New consumers can read old messages
✓ Deployment order: Schema registry → consumers → producers
What's Included
- SKILL.md: Complete Kafka architecture framework with decision trees for partitioning, serialization, and failure recovery
- Pipeline design template: Configuration scaffold for producers, brokers, consumers with tuning parameters
- Partition strategy worksheet: Cardinality analysis, throughput estimation, and rebalancing guide
- Lag monitoring checklist: Metrics to track, alerting thresholds, and consumer health dashboard template
- Troubleshooting decision tree: Root cause analysis for common issues (slow consumers, broker saturation, rebalancing loops)
- Multi-cluster playbook: Cross-region setup, failover procedures, and testing strategies
- Performance tuning reference: All broker, producer, and consumer configs with impact analysis
Who It's For
- Data engineers architecting streaming data pipelines
- Platform engineers designing multi-tenant Kafka infrastructure
- DevOps engineers optimizing Kafka cluster performance
- Backend engineers troubleshooting production consumer lag
- Streaming architects designing disaster recovery and multi-region systems
Best For
- Designing partition and replication strategies for new pipelines
- Diagnosing and fixing consumer lag and throughput issues
- Planning capacity for projected message volumes
- Validating schema evolution and backward compatibility
- Migrating or rebalancing topics across clusters
You might also like
You can systematically diagnose database performance bottlenecks by sharing your schema, slow query logs, and execution plans with Claude. It identifies root causes—missing indexes, inefficient joins, lock contention—and provides prioritized recommendations with ready-to-implement SQL. Skip the manual log analysis and get tuning strategies tailored to your workload.
Use Claude to systematically analyze your database queries, execution plans, and schema to identify performance bottlenecks. The skill generates actionable optimization recommendations with SQL rewrites, index strategies, and configuration tuning. You'll receive detailed before-and-after performance analysis to validate improvements and prioritize work by impact.
Rapidly analyze firmware logs and diagnose hardware issues that cause device failures, connectivity problems, and performance degradation. You'll identify root causes from stack traces, crash dumps, and sensor data, then generate specific optimization recommendations. This skill transforms raw device logs into actionable debugging plans that reduce time-to-resolution from hours to minutes.
You'll develop systematic approaches to injectable formulation design, from API selection through sterilization strategy. Claude helps you troubleshoot failed batches by analyzing root causes, recommends regulatory pathways (505(b)(2), ANDA, NDA), and provides science-backed solutions for stability, compatibility, and manufacturability challenges.
You'll design and implement mobile features with architectural rigor, cross-platform considerations, and edge-case handling built-in. This skill generates complete system designs, platform-specific implementation strategies, performance optimization approaches, and testing frameworks. The output is production-ready guidance spanning iOS and Android with security, offline resilience, and deployment strategies included.
Claude analyzes your code diffs and generates standardized commit messages that follow the Conventional Commits specification. The skill automatically determines the correct commit type, scope, and description based on the changes you've made, ensuring your messages are parseable by automation tools while remaining human-readable for code reviewers.
You can transform raw text into structured, labeled datasets for machine learning, analysis, and research. This skill performs named entity recognition, sentiment classification, part-of-speech tagging, and dependency parsing—generating consistent, validated annotations at scale. Use it to prepare corpora, extract entities, classify documents, or perform linguistic analysis without manual annotation.
You can architect multi-node ROS control systems from scratch, including node design patterns, communication flows, and real-time constraints. You'll debug complex node interactions using publisher/subscriber analysis, service call tracing, and action server diagnostics. You can optimize motion controllers through PID tuning, trajectory planning validation, and performance profiling to achieve precise, responsive robotic behavior.