Preventing Kafka Configuration Drift with GitOps Before Multi-Team Topic Changes Reach Production

A Kafka proof of concept usually starts with a few shell commands. One developer creates a topic, another registers a schema, and a third changes retention through a graphical administration tool. The platform works because the same people remember why every setting exists.

That approach breaks when several teams begin sharing the cluster. Development, user acceptance testing, and production slowly diverge. A topic has different partition counts in each environment. A schema is registered automatically by the first producer that starts. An access control rule is added manually during an incident and never appears in the deployment repository. Performance testing uses a topic that does not resemble production.

The visible failure may be a producer timeout or consumer lag, but the underlying problem is configuration drift. The deployed Kafka platform no longer has one reviewable definition. Its real state is scattered across scripts, user interfaces, application properties, and operator memory.

This tutorial builds a controlled delivery workflow for a multi-team customer-event platform. The goal is to move from ad hoc topic creation to requirements-driven, version-controlled Kafka resources that can be reviewed, promoted, tested, and reproduced across environments.

The Platform That Is Starting to Drift

Assume an organization is building a customer operations platform with these services:

• AccountService publishes account and contact changes.
• PurchaseService publishes completed purchase events.
• CustomerViewService consumes both streams and builds a searchable customer view.
• NotificationService reacts to selected account and purchase events.
• Kafka Connect moves approved data to analytical storage.
• Schema Registry manages message schemas.
• Platform engineers manage topics, access control lists, quotas, and cluster policies.

The early workflow looks like this:

Developer request
      |
      v
Administrator runs CLI command or opens UI
      |
      v
Topic, schema, connector, or ACL changes immediately
      |
      v
No reviewed source of truth
      |
      v
Development, UAT, and production drift apart

The target workflow is different:

Business workflow
      |
      v
Event catalogue and ownership
      |
      v
Technical topic contract
      |
      v
Nonfunctional requirements
      |
      v
Version-controlled Kafka resources
      |
      v
Automated validation and tests
      |
      v
Environment promotion
      |
      v
Reconciliation with the actual cluster

This process is not bureaucracy around Kafka. It is how the team proves why a topic exists, how it must behave, and whether the deployed configuration still matches that intent.

Step 1: Start with the Business Workflow, Not the Topic Name

A request such as "create a Kafka topic for customer updates" is incomplete. Before discussing partitions or retention, identify the business state changes that other systems need to observe.

Suppose the customer platform supports a purchase flow:

1. A customer submits an order.
2. The commerce service validates it.
3. Payment is confirmed or rejected.
4. Fulfillment reserves stock.
5. The customer receives a notification.

The event catalogue should describe meaningful facts in past tense:

• OrderSubmitted
• PaymentConfirmed
• PaymentRejected
• InventoryReserved
• OrderReadyForDispatch

For every event, record at least:

For example:

Event: PaymentConfirmed
Producer: PaymentService
Consumers: FulfillmentService, NotificationService, CustomerViewService
Meaning: Payment for an accepted order completed successfully
Trigger: The payment provider confirms the charge
Expected reaction: Reserve stock, inform the customer, update the customer view
Business owner: Commerce payments team
Data: orderId, customerId, paymentReference, confirmedAt

This catalogue is still technology-neutral. It can be reviewed by business owners, architects, and service teams before Kafka-specific implementation decisions hide the original intent.

Step 2: Decide Whether Kafka Fits the Workflow

Not every event-looking requirement belongs in Kafka. Use a short architecture intake before provisioning anything.

Ask:

• Must related events be processed in order?
• Can the workflow be partitioned without breaking business rules?
• Which field should keep related records in one partition?
• Must events remain available permanently, or only for a recovery window?
• Is the data critical enough that no acknowledged record may be lost?
• Does the payload contain personal or regulated data?
• Can consumers safely process a duplicate event?
• Is low-latency processing required, or would a scheduled batch be sufficient?
• Must several events be correlated through a shared identifier?
• Is the schema stable, optional, or expected to evolve frequently?

These questions translate business constraints into Kafka design consequences.

For example:

Rejecting an unsuitable use case early is cheaper than operating a Kafka workflow that fights the platform's semantics.

Step 3: Convert the Event into a Topic Contract

Once Kafka is selected, create a technical contract that describes both the event and its operational behavior.

The contract should identify:

• Topic name
• Producing service
• Consuming services and group identifiers
• Key field and key schema
• Value schema and serialization format
• Required headers
• Whether null keys are allowed
• Partition count
• Replication factor
• Retention or compaction policy
• Producer acknowledgment recommendation
• Consumer offset-commit recommendation
• Schema compatibility mode
• Access control requirements
• Dead-letter handling policy
• Ownership and escalation contacts

A declarative topic definition could look like this:

apiVersion: kafka.example.io/v1
kind: KafkaTopic
metadata:
  name: payment-confirmed
spec:
  topicName: commerce.payment-confirmed
  partitions: ${PAYMENT_TOPIC_PARTITIONS}
  replicas: ${PAYMENT_TOPIC_REPLICAS}
  configuration:
    cleanup.policy: delete
    retention.ms: ${PAYMENT_RETENTION_MS}
    min.insync.replicas: ${PAYMENT_MIN_ISR}
  contract:
    producer: payment-service
    keyField: orderId
    valueFormat: protobuf
    schemaCompatibility: backward-transitive
    requiredHeaders:
      - correlation-id
      - event-version

The exact custom resource format depends on the operator or infrastructure tool. The important idea is that the desired state is explicit, reviewable, and parameterized rather than hidden in a one-time command.

Include technical topics in governance

Not every Kafka topic maps directly to a business event. Kafka Streams and other processing frameworks can create repartition and changelog topics. Connectors and platform components may also create technical topics.

These topics still affect:

• Storage
• Security
• Monitoring
• Compliance
• Disaster recovery
• Capacity planning

Do not omit them from the catalogue merely because a business analyst did not name them.

Step 4: Turn Nonfunctional Requirements into Measurable Inputs

A topic contract is incomplete without performance, reliability, and recovery expectations.

Kafka settings are interconnected. Message size affects batching and memory. Throughput affects partitioning and broker capacity. Latency targets affect linger time, batch size, and consumer processing. Retention affects disk use and cloud cost.

Record message-size characteristics

Collect:

• Typical serialized message size
• Largest expected message size
• Payload shape and repetition
• Expected compression ratio
• CPU cost of the chosen compression strategy

The maximum must be considered across the entire path:

Producer limit
      |
      v
Broker and topic limit
      |
      v
Replica fetch path
      |
      v
Consumer fetch limit

Raising one limit while leaving another unchanged produces confusing runtime failures.

Do not estimate message size only from a source-language object. Measure the serialized record with realistic keys, headers, and payloads.

Separate the latency stages

"Kafka latency" can refer to several different delays:

End-to-end latency is the most useful business measure, but component-level measurements are required to locate the bottleneck.

Use percentile requirements rather than only an average. An acceptable average can hide a severe tail where a smaller group of events waits much longer.

Define normal, peak, and burst throughput

Record more than a daily total:

• Expected events per time unit
• Peak production rate
• Peak consumption rate
• Time-of-day or calendar patterns
• Burst duration
• Maximum acceptable backlog
• Time allowed to clear the backlog
• Expected growth from new channels or consumers

A structured requirement can be expressed without guessing configuration:

Steady behavior:
  Continuous order events during normal business activity

Peak behavior:
  Promotional and month-end bursts

Recovery objective:
  The consumer group must clear an accepted backlog inside the agreed recovery window

Growth:
  Re-evaluate partitions and broker resources before onboarding each major source

Performance tests will later convert these statements into evidence.

Classify data-loss tolerance

A practical classification is:

• Critical: An acknowledged event must be durably preserved.
• Important: Occasional loss may be tolerated only when a documented fallback exists.
• Non-critical: Best-effort delivery is acceptable and loss has limited impact.

This classification influences replication, acknowledgments, retry behavior, and consumer commits.

Define repeated-failure handling

Specify what happens when a consumer repeatedly fails on one record:

• Should the record be retried?
• When does retry stop?
• Is a dead-letter topic required?
• Who owns investigation and replay?
• Which metadata must be preserved?
• Which alerts must fire?
• Can later records continue processing?

A dead-letter topic without ownership and monitoring only hides the failure.

Step 5: Disable Accidental Production Provisioning

Kafka can create a missing topic when a producer first references it if automatic topic creation is enabled. Schema-aware producers can also register a previously unseen schema automatically.

These features are convenient in a personal development environment. They are dangerous in a shared production environment.

An automatically created topic inherits broker defaults. Those defaults may not match the event's requirements for:

• Partitions
• Replicas
• Retention
• Compaction
• Minimum in-sync replicas
• Access control
• Naming conventions

An automatically registered schema can make an unreviewed application build define the production contract.

For controlled environments, disable automatic creation and registration:

auto.create.topics.enable=false

auto.register.schemas=false

Provision topics and schemas before the application starts producing. A deployment should fail clearly when required infrastructure is missing instead of creating an incorrect artifact as a side effect.

Step 6: Replace Manual Changes with a GitOps Flow

GitOps uses a version-controlled repository as the desired state. A reconciliation tool compares that desired state with the deployed platform and applies approved differences.

For Kafka running on Kubernetes, the flow can use custom resources, a GitOps controller such as Argo CD or Flux, and a Kafka operator.

Engineer opens pull request
          |
          v
Review topic, schema, connector, and ACL changes
          |
          v
Merge approved desired state into Git
          |
          v
GitOps controller detects repository change
          |
          v
Kafka operator applies resource changes
          |
          v
Reconciliation reports actual status

Operators such as Strimzi or Confluent for Kubernetes can translate custom resources into Kafka artifacts.

Kafka outside Kubernetes can follow the same principle through tools such as Terraform, Ansible, Puppet, or a custom automation service using Kafka administrative APIs.

A repository structure that keeps intent visible

kafka-platform/
  catalogue/
    business-events/
    technical-topics/
    ownership/
  resources/
    base/
      topics/
      schemas/
      connectors/
      access/
    environments/
      dev/
      uat/
      prod/
  tests/
    contracts/
    integration/
    performance/
  policies/
    naming/
    retention/
    compatibility/
    security/

The base directory describes shared intent. Environment directories contain only justified differences.

A pull request should answer:

• Which business workflow requires the change?
• Who owns the event?
• Which consumers are affected?
• Does the key or partition strategy change?
• Is the schema compatible?
• Does retention increase storage cost?
• Does the change alter access permissions?
• Which tests prove the change?
• Is rollback possible?

Step 7: Use Administrative APIs Behind Automation

Kafka's Admin API can create and modify topics, inspect consumer groups, manage access rules, and automate other cluster operations. Schema Registry and Kafka Connect expose their own administrative interfaces.

A custom provisioning service can read an approved specification and apply it programmatically:

void ensureTopic(Admin admin, TopicSpecification specification)
        throws Exception {

    NewTopic desiredTopic = new NewTopic(
            specification.name(),
            specification.partitionCount(),
            specification.replicationFactor()
    );

    desiredTopic.configs(specification.kafkaConfiguration());

    admin.createTopics(List.of(desiredTopic))
            .all()
            .get();
}

A production implementation must be idempotent. It should inspect existing resources, calculate safe differences, reject unsupported destructive changes, and report drift rather than blindly running create operations.

The automation boundary should also include:

• Schema registration and compatibility checks
• Connector creation, update, pause, resume, and restart
• Connector and task status
• Access control management
• Consumer-group inspection and controlled offset operations

Do not place unrestricted administrative credentials inside ordinary application services. Provisioning automation should use a dedicated, auditable identity.

Step 8: Keep Environments Separate and Reproducible

A mature delivery process commonly uses:

• Development
• User acceptance testing, or UAT
• Production

Additional environments may include:

• Preproduction or staging
• Dedicated performance testing
• Controlled production-data troubleshooting

Separate Kafka clusters per environment provide the clearest isolation. Each environment includes its own brokers, controllers, Schema Registry, Kafka Connect, credentials, and supporting services.

The clusters do not need identical capacity. Development may use fewer brokers and a lower replication factor. Production may use stronger durability and security. Those differences must be declared, not introduced manually.

An environment matrix makes differences reviewable:

Sharing a lower-environment cluster

Some organizations share one cluster for development and UAT to reduce cost. This requires strict separation:

• Environment prefixes in topic names
• Unique credentials per environment
• ACLs that prevent cross-environment access
• Separate Schema Registry environments to avoid contract conflicts
• Environment-specific configuration variables
• Quotas that prevent one test from consuming all shared resources

The operational burden of this isolation can outweigh the infrastructure savings.

Do not stream raw production data into UAT

UAT may run a newer schema than production. Raw production events can fail against that schema. Customer data can also expose personal or confidential information.

Use a controlled migration pipeline:

1. Select only required records.
2. Mask or anonymize sensitive fields.
3. Transform data into the UAT-compatible contract.
4. Transfer it through an approved process.
5. Apply retention and access controls in the destination.

Step 9: Build Test Gates Around the Kafka Change

A topic that can be deployed is not necessarily ready for production. Test at three levels.

Unit Tests: Prove Application Behavior in Isolation

A producer unit test should verify that application logic passes the correct event to its producer abstraction.

@Test
void publishesPaymentConfirmationAfterSuccessfulCharge() {
    PaymentPublisher publisher = mock(PaymentPublisher.class);
    PaymentApplication service = new PaymentApplication(publisher);

    service.confirmPayment(sampleConfirmation());

    verify(publisher).publish(
            argThat(event ->
                    event.orderId().equals(expectedOrderId())
                    && event.customerId().equals(expectedCustomerId()))
    );
}

This test does not prove network connectivity, serialization, topic existence, or Kafka acknowledgments. That is intentional. It provides fast feedback about application behavior.

A consumer unit test should call the handler directly with a deserialized event and verify the business side effect:

@Test
void updatesCustomerViewFromPaymentEvent() {
    CustomerViewRepository repository = mock(CustomerViewRepository.class);
    PaymentEventHandler handler = new PaymentEventHandler(repository);

    handler.handle(samplePaymentConfirmed());

    verify(repository).recordConfirmedPayment(
            expectedCustomerId(),
            expectedOrderId()
    );
}

Unit tests should assert observable behavior rather than private implementation details.

Integration Tests: Prove the Real Kafka Boundary

Integration tests should use a real broker process. Testcontainers can start Kafka inside Docker for the test run.

Test starts Dockerized Kafka
        |
        v
Test creates required topic and schema
        |
        v
Application uses real producer or consumer client
        |
        v
Kafka persists and delivers the record
        |
        v
Test verifies output or side effect

Producer integration tests should prove:

• Topic resolution
• Serialization
• Actual send and acknowledgment behavior
• Headers and key selection
• Schema Registry integration when used
• Security configuration on critical paths

Consumer integration tests should prove:

• Deserialization
• Group subscription
• Offset behavior
• Handler invocation
• Failure and retry policy
• Rebalance-sensitive behavior where relevant
• Transaction visibility where relevant

Mock-based tests cannot prove TLS, SASL, ACLs, schema compatibility, offsets, rebalances, or Kafka transactions. Use Testcontainers for the paths where those guarantees matter.

Performance Tests: Prove the Nonfunctional Requirements

Kafka includes command-line tools for producer, consumer, and end-to-end latency testing.

A test plan can invoke:

kafka-producer-perf-test.sh \
  --topic performance.payment-events \
  --num-records "${TOTAL_RECORDS}" \
  --record-size "${RECORD_BYTES}" \
  --throughput "${TARGET_RATE}" \
  --producer.config producer-performance.properties

kafka-consumer-perf-test.sh \
  --bootstrap-server "${BOOTSTRAP_SERVERS}" \
  --topic performance.payment-events \
  --messages "${TOTAL_RECORDS}"

kafka-run-class.sh kafka.tools.EndToEndLatency \
  "${BOOTSTRAP_SERVERS}" \
  performance.payment-events \
  "${MESSAGE_COUNT}" \
  "${ACK_MODE}" \
  "${MESSAGE_SIZE}"

Run performance tests against topics configured like the intended deployment:

• Representative partition count
• Representative replication factor
• Realistic retention and compaction
• Real producer and consumer properties
• Realistic message-size distribution
• Expected compression
• Normal and burst traffic patterns

Collect:

• Producer throughput
• Consumer throughput
• End-to-end throughput
• Median latency
• Tail latency percentiles
• Maximum observed latency
• Backlog growth
• Backlog recovery time
• Error and timeout rates
• Resource consumption

Tools such as JMeter, Gatling, and k6 can support repeatable parameterized scenarios. Kafka's Trogdor framework can add load generation and fault injection, including broker failures and network disruption.

A successful test should answer whether the declared requirement was met, not merely print a high throughput number.

Step 10: Turn the Workflow into CI/CD Gates

A deployment pipeline can apply these checks in order:

Pull request opened
      |
      v
Naming and ownership validation
      |
      v
Schema compatibility validation
      |
      v
Topic-policy validation
      |
      v
Unit tests
      |
      v
Integration tests with real Kafka
      |
      v
Plan environment changes
      |
      v
Review destructive or costly differences
      |
      v
Apply to target environment
      |
      v
Reconcile and verify resource health

Performance tests may run in a dedicated environment rather than on every commit. Critical changes to partitions, serializers, compression, batching, or consumer concurrency should trigger them before production promotion.

The pipeline should block:

• Missing owners
• Invalid topic names
• Unreviewed schema incompatibility
• Production auto-creation
• Replication factors unsupported by the target cluster
• Retention outside organizational policy
• Missing access rules
• Destructive topic changes without a migration plan
• Failed connector tasks
• Integration failures
• Unmet performance requirements

Common Mistakes

Creating topics from application startup

This hides infrastructure provisioning inside business services and can apply unsafe defaults.

Treating the cluster as the documentation

Inspecting Kafka tells you what exists, not why it exists or who approved it.

Recording functional requirements but not load characteristics

Partitions, buffers, compression, memory, and cost cannot be justified without message size, throughput, latency, and growth expectations.

Using the same configuration in every environment

Different broker counts and data classifications require explicit environment values.

Letting environment differences accumulate outside Git

An emergency manual change becomes permanent drift unless it is reconciled back into the source of truth.

Testing only producer and consumer classes with mocks

Mocks prove application calls, not Kafka protocols, persistence, security, group behavior, or schemas.

Running performance tests with unrealistic topics

A single-partition, low-replication test says little about a production configuration with different durability and parallelism.

Copying raw production data into UAT

Schema differences and sensitive data make uncontrolled copies unsafe.

Ignoring technical topics

Internal repartition and changelog topics still consume resources and require governance.

Creating a dead-letter topic without a recovery process

The pipeline appears healthy while failed business events accumulate unprocessed.

Production Readiness Checklist

Before promoting a Kafka workflow, confirm:

• The business process and event triggers are documented.
• Every event has a business and technical owner.
• Producers, consumers, and consumer groups are known.
• Ordering and partition-key requirements are explicit.
• The topic contract includes partitions, replicas, retention, compaction, serialization, headers, acknowledgments, commits, and compatibility.
• Technical topics are included in governance.
• Average and maximum serialized record sizes have been measured.
• Latency requirements use meaningful end-to-end and percentile metrics.
• Normal, peak, burst, recovery, and growth throughput are defined.
• Data-loss tolerance is classified.
• Duplicate handling and dead-letter recovery are defined.
• Automatic topic creation is disabled in controlled environments.
• Automatic schema registration is disabled where review is required.
• Desired state is version-controlled.
• Environment differences are parameterized and reviewed.
• Administrative automation uses a dedicated identity.
• Unit tests cover isolated producer and consumer behavior.
• Integration tests use a real Kafka broker for critical paths.
• Performance tests use representative topic and client settings.
• CI/CD blocks incompatible, destructive, insecure, or untested changes.
• The deployed cluster has been reconciled with the repository after release.

Conclusion

Kafka configuration drift begins when a shared platform has no authoritative path from business intent to deployed resources. Manual commands and user interfaces are useful for exploration, but they do not provide repeatability, review history, environment promotion, or reliable testing.

A controlled project starts with an event catalogue, converts business facts into complete topic contracts, and adds measurable requirements for size, throughput, latency, reliability, and recovery. Git becomes the desired state, while operators, infrastructure tools, or administrative APIs reconcile approved changes with each environment.

Unit tests protect application logic, integration tests prove the real Kafka boundary, and performance tests verify the nonfunctional contract. Together, these practices turn Kafka provisioning from a collection of remembered commands into an auditable delivery process that can support more teams without allowing production to become the only accurate source of configuration.