Cloud & Infrastructure: AWS-first with Kubernetes, Kafka and Docker

I have worked with AWS on every one of my projects since 2015. In my current role, the systems I share responsibility for process several hundred terabytes of data every day. This page describes how I build and run the infrastructure for that, with AWS as the shared operating context and Kafka, Kubernetes and Docker as engineering practices in their own right.

AWS-first: a deliberate architecture decision

To me, “AWS-first” is a trade-off, not a commitment to one vendor. Anyone planning infrastructure today can choose between two paths: a cross-cloud abstraction layer that keeps a later change of provider open, or the native tools of the platform the systems actually run on. My projects don’t currently need cloud independence. An extra abstraction layer would mostly add complexity without any concrete benefit.

That is why I build on AWS-native components: CloudFormation for Infrastructure as Code, CodeBuild and CodePipeline for CI/CD, Cognito for identity, and QuickSight for reporting. The decision isn’t final. It holds as long as the circumstances stay the same, and I would revisit it as soon as a real multi-cloud need comes up. There’s more on this on the page about Terraform, which I have put on hold for this reason.

AWS experience as a software architect: hands-on operations, not slideware

My AWS experience as a software architect comes from more than ten years of production use across several employers, not from a single project. The clearest evidence is the data volume in my current role: several hundred terabytes pass through the AWS data services I work with every day. At that scale, mistakes in partitioning, cost control or error handling show up quickly. You learn to build systems that hold up in long-term operation.

Several initiatives from earlier projects show what this looks like in practice: experience migrating self-hosted streaming infrastructure to managed services such as Amazon MSK (more on that in the Kafka section), building BI solutions on AWS analytics services such as Athena, Glue and QuickSight, and introducing single sign-on solutions on AWS.

All of this infrastructure is defined as code. I have used CloudFormation in client projects since 2016. Resources aren’t clicked together in the web console. They are versioned, reviewed and rolled out through the CI/CD pipeline. That makes it clear why an environment looks the way it does, and any environment can be rebuilt identically at any time.

Kafka and AWS MSK: experience with event streaming at scale

For years, Apache Kafka has been a central tool for event-driven architectures in my work, wherever it runs. I have run Kafka myself and also used it as the managed service Amazon MSK. The practice is the same in both cases: design topics sensibly, decouple data flows between producers and consumers, and shape streams so that downstream systems can process them reliably.

In earlier projects, I’ve migrated self-hosted Kafka clusters to Amazon MSK to reduce operational overhead without touching the streaming architecture. AWS then handles cluster maintenance, and the data-flow architecture stays the same. I see this as a good example of the AWS-first trade-off. Managed services take routine work off the team’s plate, while the engineering decisions stay with the team.

Kafka is a core part of the data pipelines that carry a large share of the data processed each day. Running streaming systems of this size also means you need tools for when things go wrong, for example to trace individual messages when needed.

Kubernetes in practice on Amazon EKS

I mostly use Kubernetes as Amazon EKS, building on my many years of hands-on AWS work. With EKS, AWS manages the control plane. The real work still sits with the team, and so with me: sizing node groups, integrating networking cleanly into the VPC, defining IAM roles, configuring load balancers and running the workloads themselves.

I see Kubernetes as one link in a chain. Docker provides the image, CloudFormation lays the infrastructure foundation, and Kubernetes handles orchestration: it distributes containers across nodes, keeps the desired number of instances running and manages rollouts of new versions. Each link has a clearly defined job. That makes the chain as a whole easy to understand and easier to analyze when something fails. The underlying domain-boundary practice determines how many services have to be deployed independently.

Docker containers in practice: from laptop to production

Docker is part of my everyday work, and I use it daily. I rely on containerized staging sandboxes where changes can be tried out safely before they reach a real environment. This makes development environments reproducible: anyone setting up a project from scratch gets the same environment as everyone else. I also review redesigns locally in a container before they go live.

I treat Dockerfiles and Compose files like any other code. They are versioned, shared and reviewed. That makes the environment itself part of the project instead of knowledge that lives only in a few people’s heads.

Docker is also the first step toward a platform. An image that runs locally also runs on EKS without changes. There, the cluster takes care of scaling, self-healing and rollouts. Because the same image runs on the laptop, in staging and in production, deployments bring far fewer surprises.

The common thread: independent practices, one shared operating context

To me, Kafka, Kubernetes and Docker are not AWS products. They are engineering practices in their own right, each with its own principles. Event streaming, container orchestration and reproducible runtime environments work on any platform. AWS is where I bring them together and run them: with managed services like MSK and EKS where they save operational effort, and with CloudFormation and native CI/CD tools as the foundation that ties everything together.

The AWS-first strategy keeps this stack lean. Each layer uses the tools of the platform it runs on, and there’s no extra abstraction layer to maintain. Day-to-day operations show that the approach works: these systems process several hundred terabytes every day.