Getting started with Apache Kafka

2 minute read
kafka
,

Today I started with the excellent Pluralsight course “Getting Started with Apache Kafka”.

The course is focused on using an Ubuntu test server for Kafka and a Java development environment. This blog post is a list of resources that I found helpful while exploring Apache Kafka, with a focus on the Microsoft stack.

Hosting a Kafka environment

Multiple options exist for hosting your test environment. One option is to deploy Apache Kafka in an Azure HDInsight cluster (quickstart tutorial + Azure Friday). This option seemed a bit overkill for me as I was searching a “quick start” experience.

Another option are containers. While searching for Docker images, I stumbled upon the Bitnami Kafka Stack.
They offer both Docker and VM images that you can use for running locally on your machine or for deploying to the Azure cloud (Bitnami Azure Marketplace).

But, also this still takes some time before you can connect to your Kafka test cluster. So in the end, I went for a Confluent Kafka cluster, running in Azure. It took me less then 3 minutes to register a new Confluent account, create a new Kafka cluster and connect to it with Conduktor. This solution is of course not free. Luckily, it is possible to use the Confluent platform for 3 months with a 50 USD spending credit per month.

Connecting to Kafka from .NET

When it comes to connecting to Kafka with .NET, there is only one solution you should use and that is “confluent-kafka-dotnet”. All other packages are outdated and suggest you should use that one.

As mentioned in the beginning, the Pluralsight is focusing on the Java client for Kafka. Not all concepts are the same when using the Confluent .NET client, therefore you should dig into the wiki on the GitHub project. It provides all info needed to create your first simple producer and consumer application. Also, the blogpost ‘Designing the .NET API for Apache Kafka’ gives some interesting back story.

What’s next?

After completing the ‘getting started course’ on Pluralsight I can recommend the following resources to continue your learnings: