Kafka Pipeline Architect
Design production Kafka pipelines with verified performance and scale
What You Can Do
You can architect production-ready Kafka pipelines with schema validation, partition strategies, and lag management. Claude helps you identify bottlenecks, optimize throughput, and design fault-tolerant systems with concrete performance guarantees. You'll get architecture diagrams, configuration templates, and troubleshooting playbooks for real-world streaming challenges.
Features
Balance load across brokers based on key cardinality and throughput
Validate message format, handle schema evolution, prevent data corruption
Design metrics dashboards and alert thresholds for consumer lag
Design replication, exactly-once semantics, and crash recovery workflows
Identify bottlenecks in producers, brokers, and consumers with concrete tuning parameters
Calculate broker disk, memory, network requirements from message volume projections
Design cross-region replication, disaster recovery, and migration strategies
Example Output
Example 1: Partition Strategy Analysis
Topic: user-events
Current partitions: 4
Recommended partitions: 12
Rationale: Key cardinality ~2M users, target throughput 50K msgs/sec
Partition assignment: Round-robin on user_id hash
Expected rebalance time: 2.3s
Example 2: Consumer Lag Diagnosis
Consumer group: analytics-processor
Lag by partition:
- P0: 1.2M messages (12 mins behind)
- P1: 450K messages (4.5 mins behind)
- P2: 50K messages (30 secs behind)
Root cause: Consumer thread on P0 processing at 1.6K msgs/sec vs 2.1K msgs/sec ingest
Recommendation: Increase consumer parallelism or add 2 more consumer instances
Example 3: Schema Evolution Checklist
✓ New field: user_country (string, optional)
✓ Default value: 'UNKNOWN'
✓ Backward compatibility: Old producers can read new schema
✓ Forward compatibility: New consumers can read old messages
✓ Deployment order: Schema registry → consumers → producers
What's Included
- SKILL.md: Complete Kafka architecture framework with decision trees for partitioning, serialization, and failure recovery
- Pipeline design template: Configuration scaffold for producers, brokers, consumers with tuning parameters
- Partition strategy worksheet: Cardinality analysis, throughput estimation, and rebalancing guide
- Lag monitoring checklist: Metrics to track, alerting thresholds, and consumer health dashboard template
- Troubleshooting decision tree: Root cause analysis for common issues (slow consumers, broker saturation, rebalancing loops)
- Multi-cluster playbook: Cross-region setup, failover procedures, and testing strategies
- Performance tuning reference: All broker, producer, and consumer configs with impact analysis
Who It's For
- Data engineers architecting streaming data pipelines
- Platform engineers designing multi-tenant Kafka infrastructure
- DevOps engineers optimizing Kafka cluster performance
- Backend engineers troubleshooting production consumer lag
- Streaming architects designing disaster recovery and multi-region systems
Best For
- Designing partition and replication strategies for new pipelines
- Diagnosing and fixing consumer lag and throughput issues
- Planning capacity for projected message volumes
- Validating schema evolution and backward compatibility
- Migrating or rebalancing topics across clusters
You might also like
Analyze Solidity and other smart contract code for security vulnerabilities, gas inefficiencies, and best practice violations. Get detailed reports with risk scoring, remediation suggestions, and optimization recommendations. Whether you're auditing before deployment or reviewing third-party contracts, this skill identifies critical issues faster than manual review.
You can systematically diagnose database performance bottlenecks by sharing your schema, slow query logs, and execution plans with Claude. It identifies root causes—missing indexes, inefficient joins, lock contention—and provides prioritized recommendations with ready-to-implement SQL. Skip the manual log analysis and get tuning strategies tailored to your workload.
This skill provides a structured approach to analyzing process problems, identifying root causes, and recommending capacity optimizations. You'll get clear bottleneck identification, data-driven recommendations, and a framework to validate whether your solutions actually work. Perfect for diagnosing why workflows are slow and finding the leverage points that matter most.
Use Claude to systematically analyze your database queries, execution plans, and schema to identify performance bottlenecks. The skill generates actionable optimization recommendations with SQL rewrites, index strategies, and configuration tuning. You'll receive detailed before-and-after performance analysis to validate improvements and prioritize work by impact.
You'll design and implement mobile features with architectural rigor, cross-platform considerations, and edge-case handling built-in. This skill generates complete system designs, platform-specific implementation strategies, performance optimization approaches, and testing frameworks. The output is production-ready guidance spanning iOS and Android with security, offline resilience, and deployment strategies included.
Design and optimize injectable formulations by analyzing your active pharmaceutical ingredient (API), selecting compatible excipients, and predicting stability outcomes. You'll receive systematic workflows that guide you through API characterization, formulation architecture, and risk mitigation—enabling faster development cycles and regulatory-ready documentation.
Claude analyzes your code diffs and generates standardized commit messages that follow the Conventional Commits specification. The skill automatically determines the correct commit type, scope, and description based on the changes you've made, ensuring your messages are parseable by automation tools while remaining human-readable for code reviewers.
You can architect multi-node ROS control systems from scratch, including node design patterns, communication flows, and real-time constraints. You'll debug complex node interactions using publisher/subscriber analysis, service call tracing, and action server diagnostics. You can optimize motion controllers through PID tuning, trajectory planning validation, and performance profiling to achieve precise, responsive robotic behavior.