Building Real Integration Tests in Go with Testcontainers

Unit tests are excellent for checking pure logic. They are fast, isolated, and easy to run after every small change. But an enterprise Go service rarely runs as pure logic. It writes to databases, reads from caches, publishes messages, and depends on infrastructure that can fail in ways a mock will never expose.

A mock can tell you that your repository called Save. It cannot prove that your database driver accepts the connection string, that your schema matches the code, that a column type is compatible with your struct field, or that the service is really ready when the test starts.

That is where integration tests become useful. An integration test checks several real parts of the system together. In this post, the focus is a common Go backend scenario: testing a repository against a real MySQL database started in a Docker container by Testcontainers.

The Problem

Imagine a small dispatch service. Its job is to record which vehicle should serve which route. The business code may be simple, but the persistence boundary is not completely under your control. It depends on several moving parts:

• Go code that uses database/sql
• A MySQL driver
• A database schema
• Connection details such as host, port, user, password, and database name
• Docker networking during tests
• A CI runner that can start containers

The expected output of the test is also clear: when the repository saves a dispatch record, the same record should be readable from the real database through the same repository API.

A unit test can check that your service calls a repository interface correctly. An integration test checks whether the repository actually works with the infrastructure it claims to support.

Go test process
  |
  v
Testcontainers helper
  |
  v
Docker starts MySQL
  |
  v
Repository uses database/sql
  |
  v
Test verifies real save and read behavior

This does not mean every test should use a real database. The goal is balance. Keep most tests fast and isolated, then add focused integration tests around the places where your code crosses a real boundary.

Unit Tests and Integration Tests Solve Different Problems

A healthy Go project uses both. They are not competitors.

A repository backed only by mocks can create false confidence. The mock may accept a field name that the database does not have. It may return a value that the real driver cannot scan. It may ignore timing and readiness problems that happen in CI.

Integration tests give you a controlled way to catch those failures before deployment.

What Testcontainers Does

Testcontainers is a testing library that starts real services in Docker containers during test execution. In Go projects, testcontainers-go lets a test start dependencies such as MySQL, PostgreSQL, Redis, Kafka, RabbitMQ, and other services without asking developers to run them manually.

The workflow is simple:

1. The Go test starts.
2. Testcontainers asks Docker to create the required service container.
3. The test waits until the service is actually ready.
4. The test connects to the service using mapped host and port values.
5. The test runs real application code against that service.
6. Testcontainers cleanup handles the container lifecycle.

The important part is readiness. A container can be running while the service inside it is still starting. Fixed sleeps make tests flaky because different machines and CI runners start containers at different speeds. A wait strategy tells the test to continue only when the service is ready enough to accept work.

Choosing the Tool

For Go projects, two common container-based testing options are testcontainers-go and ory/dockertest.

testcontainers-go is usually the stronger default for enterprise work because it provides ready-made modules for common services, wait strategies, cleanup through the Testcontainers reaper process, and patterns that match the broader Testcontainers ecosystem.

dockertest is smaller and easier to understand, but it gives you fewer high-level features. Cleanup is more manual, and you do more of the service readiness work yourself.

A practical rule is simple: use testcontainers-go unless your needs are extremely small and you intentionally want a lower-level Docker wrapper.

Recommended Project Layout

Keep container setup separate from repository tests. The test should describe behavior, not Docker plumbing.

fleet-service/
  internal/
    dispatch/
      domain.go
      repository/
        mysql_store.go
        mysql_store_integration_test.go
  test_containers/
    mysql.go
  go.mod
  go.sum

The internal/dispatch/repository package owns the repository implementation and its tests. The test_containers folder owns reusable test infrastructure. If the project later adds Redis, Kafka, or PostgreSQL, you can add one small helper file per service instead of copying setup code into every test.

Keep Integration Tests Out of the Default Test Run

Most developers run this command frequently:

go test ./...

That command should stay fast. If integration tests start containers every time, developers may stop running the full test suite locally. Build tags solve this problem.

A Go build tag is a compile-time condition. A file with this line is ignored unless the matching tag is passed to the go test command:

//go:build integration_test

package repository_test

Now the normal command skips integration tests:

go test ./...

And this command includes them:

go test -timeout 90s -tags=integration_test ./...

This separation gives the team two feedback loops:

• Fast feedback from unit tests during normal development
• Real dependency feedback from integration tests when the team asks for it or when CI runs the integration stage

Add the Test Dependencies

Start with the packages needed for Docker-based tests and MySQL access.

go get github.com/testcontainers/testcontainers-go@latest
go get github.com/docker/go-connections@latest
go get github.com/go-sql-driver/mysql@latest

After these commands, commit both go.mod and go.sum. The module file records what your project requires, and the checksum file helps keep dependency resolution reproducible across machines.

Pattern: One Container, Many Tests

Starting MySQL can take several seconds. Starting a new MySQL container for every test is usually wasteful. A better pattern is to start one container once and reuse its host and port across all integration tests in the same package run.

The standard library type sync.Once is perfect for this. It guarantees that setup code runs only one time, even if several tests ask for the container.

The container helper below uses these ideas:

• A build tag keeps the helper out of normal unit test runs.
• A fixed MySQL image version avoids surprise changes.
• Docker chooses the host port dynamically to avoid conflicts.
• A wait strategy waits for MySQL readiness instead of using a fixed sleep.
• The helper returns host and port so tests do not need to know Docker details.

//go:build integration_test

package test_containers

import (
    "context"
    "fmt"
    "sync"
    "time"

    "github.com/docker/go-connections/nat"
    "github.com/testcontainers/testcontainers-go"
    "github.com/testcontainers/testcontainers-go/wait"
)

var (
    mysqlOnce     sync.Once
    mysqlInstance testcontainers.Container
    mysqlHost     string
    mysqlPort     string
    mysqlErr      error
)

func MySQL(ctx context.Context, dbName, user, pass string) (string, string, error) {
    mysqlOnce.Do(func() {
        servicePort := nat.Port("3306/tcp")

        request := testcontainers.ContainerRequest{
            Image:        "mysql:8.0.44",
            ExposedPorts: []string{string(servicePort)},
            Env: map[string]string{
                "MYSQL_DATABASE":      dbName,
                "MYSQL_USER":          user,
                "MYSQL_PASSWORD":      pass,
                "MYSQL_ROOT_PASSWORD": pass + "-root",
            },
            WaitingFor: wait.ForListeningPort(servicePort).
                WithStartupTimeout(90 * time.Second),
        }

        mysqlInstance, mysqlErr = testcontainers.GenericContainer(
            ctx,
            testcontainers.GenericContainerRequest{
                ContainerRequest: request,
                Started:          true,
            },
        )
        if mysqlErr != nil {
            mysqlErr = fmt.Errorf("start mysql container: %w", mysqlErr)
            return
        }

        mysqlHost, mysqlErr = mysqlInstance.Host(ctx)
        if mysqlErr != nil {
            mysqlErr = fmt.Errorf("read mysql host: %w", mysqlErr)
            return
        }

        mappedPort, err := mysqlInstance.MappedPort(ctx, servicePort)
        if err != nil {
            mysqlErr = fmt.Errorf("read mysql mapped port: %w", err)
            return
        }

        mysqlPort = mappedPort.Port()
    })

    return mysqlHost, mysqlPort, mysqlErr
}

This code intentionally avoids binding MySQL to a fixed host port. Fixed ports are convenient at first, but they cause conflicts when multiple test processes run on the same machine or when CI runs jobs in parallel. Let Docker choose the host port, then ask Testcontainers which port was mapped.

Write a Repository Test Against the Real Database

The test should still read like a behavior test. It should not be a long Docker script. The helper starts MySQL; the repository test connects, prepares storage, writes a record, and reads it back.

The domain names below are deliberately small. A dispatch record represents one vehicle assigned to one route.

//go:build integration_test

package repository_test

import (
    "context"
    "database/sql"
    "fmt"
    "testing"
    "time"

    _ "github.com/go-sql-driver/mysql"

    "example.com/fleet/internal/dispatch"
    "example.com/fleet/internal/dispatch/repository"
    "example.com/fleet/test_containers"
)

func TestMySQLDispatchStore_SaveThenFind(t *testing.T) {
    ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
    defer cancel()

    host, port, err := test_containers.MySQL(ctx, "dispatch_test", "tester", "secret")
    if err != nil {
        t.Fatalf("mysql test container is not ready: %v", err)
    }

    dsn := fmt.Sprintf(
        "tester:secret@tcp(%s:%s)/dispatch_test?parseTime=true",
        host,
        port,
    )

    db, err := sql.Open("mysql", dsn)
    if err != nil {
        t.Fatalf("open database handle: %v", err)
    }
    defer db.Close()

    if err := db.PingContext(ctx); err != nil {
        t.Fatalf("ping database: %v", err)
    }

    prepareDispatchStorage(t, ctx, db)
    resetDispatchStorage(t, ctx, db)

    store := repository.NewMySQLDispatchStore(db)

    expected := dispatch.Dispatch{
        ID:          "dispatch-1001",
        VehicleCode: "bus-21",
        RouteCode:   "route-5",
        State:       "scheduled",
        CreatedAt:   time.Now().UTC().Truncate(time.Second),
    }

    if err := store.Save(ctx, expected); err != nil {
        t.Fatalf("save dispatch: %v", err)
    }

    got, err := store.FindByID(ctx, expected.ID)
    if err != nil {
        t.Fatalf("find dispatch by id: %v", err)
    }

    if got.ID != expected.ID {
        t.Errorf("id mismatch: got %q, want %q", got.ID, expected.ID)
    }
    if got.VehicleCode != expected.VehicleCode {
        t.Errorf("vehicle mismatch: got %q, want %q", got.VehicleCode, expected.VehicleCode)
    }
    if got.RouteCode != expected.RouteCode {
        t.Errorf("route mismatch: got %q, want %q", got.RouteCode, expected.RouteCode)
    }
}

func prepareDispatchStorage(t *testing.T, ctx context.Context, db *sql.DB) {
    t.Helper()
    // Run your normal migration or schema bootstrap path here.
    // Keep schema setup in one reusable helper instead of spreading it across tests.
}

func resetDispatchStorage(t *testing.T, ctx context.Context, db *sql.DB) {
    t.Helper()
    // Clear rows created by earlier tests, restore a snapshot, or create an isolated schema.
}

The storage setup helpers are intentionally centralized. In a real project, they should call the same migration or schema bootstrap path that the application trusts. Do not scatter setup details across many tests. When schema setup lives in one place, tests are easier to maintain and less likely to drift away from production behavior.

Why Fatalf and Errorf Are Used Differently

The test uses t.Fatalf for infrastructure failures because the test cannot continue safely when MySQL is unavailable, the database handle cannot be opened, or saving the record fails. Continuing after those failures would create confusing secondary errors.

Use t.Errorf when the test can still collect more useful information. In the example, once a record has been loaded, multiple fields can be compared. If the vehicle code and route code are both wrong, two Errorf calls show both problems in one test run.

A simple rule works well:

• Use Fatalf when the next line would be meaningless or unsafe.
• Use Errorf when the test can continue checking more conditions.

Running the Tests

Run fast tests by default:

go test ./...

Run integration tests explicitly:

go test -timeout 90s -tags=integration_test ./...

Run one focused integration test while debugging:

go test -run TestMySQLDispatchStore_SaveThenFind -timeout 90s -tags=integration_test ./internal/dispatch/repository

The timeout matters because containers can take time to start, especially on busy CI workers. A clear timeout is better than a test process that hangs indefinitely.

Testing More Than Save and Find

A single save-and-find test proves the basic path, but a repository usually has more responsibilities. Add integration tests where the real database contract matters.

Useful cases include:

1. Save several dispatch records with different states, then verify that the repository can return only the requested state.
2. Check how the repository behaves when a record does not exist.
3. Check that date and time values survive the write and read path correctly.
4. Check that duplicate identifiers or invalid persistence states produce the error behavior your application expects.
5. Check that the repository uses context cancellation correctly when an operation cannot complete.

The goal is not to test every line through a container. The goal is to test the contract between your Go code and the real service.

Database State: The Hard Part of Shared Containers

Reusing one container is fast, but it introduces a problem: state can leak from one test to another.

There are several ways to handle this:

• Reset the affected tables before each test.
• Create a unique database or schema for each test group.
• Use Testcontainers snapshot support when the setup is expensive and many tests need a clean baseline.
• Use a dedicated container for tests that cannot safely share state.

The best choice depends on cost and isolation. If the schema is small, resetting state may be enough. If the service has complex setup, a snapshot can save time. If tests mutate global service configuration or run in heavy parallel mode, dedicated containers may be safer.

A reliable integration test must be repeatable. It should pass when run alone, after another test, or in a different order.

Parallel Integration Tests

Go can run tests in parallel, but parallelism is only safe when tests do not fight over shared resources.

Inside a test, t.Parallel() marks the test as safe to run beside other parallel tests:

func TestMySQLDispatchStore_FindScheduled(t *testing.T) {
    t.Parallel()

    // Use isolated data, a unique schema, a restored snapshot, or a dedicated container.
}

Across packages, the -p flag controls how many packages can be tested at the same time:

go test ./... -timeout 90s -tags=integration_test -p 4

Parallel execution can reduce CI runtime, especially when independent packages start independent containers. But it can also create flaky tests if several tests write to the same database rows, bind the same host ports, or reuse the same files.

Use parallel integration tests only when isolation is designed, not assumed.

Extending the Pattern to Redis, Kafka, or PostgreSQL

The same pattern works for other infrastructure:

1. Create one helper file per service under test_containers.
2. Put container startup details in that helper.
3. Use sync.Once when one shared container is enough.
4. Return only what tests need, such as host, port, connection string, or client configuration.
5. Keep service cleanup and readiness logic out of business tests.

For example, a caching test might start Redis and MySQL, write through the repository, read through the cache path, and verify that both layers agree. A messaging test might start Kafka, publish an event, and verify that a consumer reacts as expected.

The structure stays the same even when the service changes:

Test behavior
  |
  v
Service-specific container helper
  |
  v
Testcontainers starts real dependency
  |
  v
Application code talks to dependency through normal adapters

This keeps tests readable as the system grows.

CI Integration

Local integration tests are helpful, but CI is where they become a team safety net. CI means continuous integration: the pipeline builds and tests code automatically after changes are pushed.

A container-based test stage needs a little more care than a unit test stage. The runner must be able to start Docker containers, pull images, and wait for services. When that setup is missing, tests fail because the environment is wrong, not because the code is wrong.

A practical pipeline shape looks like this:

Stage 1: fast checks
  - format and static checks
  - go test ./...

Stage 2: integration checks
  - make Docker available to the runner
  - pull or reuse service images
  - go test -timeout 90s -tags=integration_test ./...

Stage 3: build or package
  - run only after fast and integration checks pass

Keep integration tests in their own stage. If unit tests already fail, starting containers wastes time. Separating stages also makes failure reports clearer: fast logic failure and real dependency failure are different signals.

CI Practices That Prevent Flaky Tests

Container-based tests are more realistic, but they can become unstable if the pipeline is careless.

Use these practices:

• Make sure Docker or a compatible container runtime is available in the CI runner.
• Pin service image versions instead of using latest.
• Cache image layers where your CI platform supports it.
• Use wait strategies instead of fixed sleeps.
• Set a test timeout that gives containers enough time to start but still prevents endless hangs.
• Keep integration tests in a separate pipeline stage.
• Control parallelism so tests do not share unsafe state.
• Prefer dynamic mapped ports instead of fixed host ports.
• Reset database state, restore snapshots, or isolate schemas before each test.

A good CI setup makes integration tests boring. They should fail when the code, schema, or service contract is wrong, not because the pipeline randomly started MySQL too slowly.

Common Mistakes

Mistake 1: Replacing Unit Tests with Integration Tests

Integration tests are not a better version of unit tests. They are slower and harder to debug. Keep pure business rules in unit tests. Use integration tests for boundaries such as databases, caches, brokers, and real adapters.

Mistake 2: Running Containers in Every Test

Starting a container per test is simple but expensive. Use one shared container when tests can safely share the service. Use dedicated containers only when isolation is more important than speed.

Mistake 3: Depending on Fixed Sleep Times

A fixed sleep may work on your laptop and fail in CI. Use Testcontainers wait strategies so tests continue when the service is actually ready.

Mistake 4: Using Floating Host Ports Manually

Hard-coded host ports create conflicts. Let Docker choose the host port and ask Testcontainers for the mapped value.

Mistake 5: Sharing State Without a Reset Strategy

A test that passes only when it runs first is not reliable. Every integration test should control its starting state.

Mistake 6: Hiding Integration Tests Inside the Normal Test Run

If go test ./... becomes too slow, developers run it less often. Use build tags so fast tests remain fast and integration tests stay available on demand.

Mistake 7: Skipping Integration Tests in CI

If integration tests run only on one developer's machine, the team loses the main value. Put them in CI so every important change is checked against real dependencies.

Practical Checklist

Before calling a Go integration test reliable, check these points:

• The file has the integration_test build tag.
• The normal go test ./... command remains fast.
• Testcontainers starts the real dependency automatically.
• The container uses a fixed image version.
• Readiness is handled by a wait strategy, not a sleep.
• Host and port are discovered from Testcontainers.
• The test uses context.Context with a timeout.
• Infrastructure failures use Fatalf.
• Field comparison failures use Errorf when more checks can continue.
• Database state is reset, isolated, or restored before each test.
• CI has a dedicated integration test stage.
• Parallelism is enabled only when resources are isolated.

Conclusion

Integration tests make Go services prove themselves against real infrastructure. They do not replace unit tests, but they close the gap that mocks leave open: driver behavior, schema compatibility, service readiness, Docker networking, and CI environment reality.

A strong pattern keeps the suite practical. Put container setup in reusable helpers, start expensive services once with sync.Once, protect integration files with build tags, use wait strategies, and run the slower tests in a dedicated CI stage. With that structure, developers keep fast local feedback while the project still verifies the real contracts that production depends on.