Building Resilient Event-Driven Architectures (EDA) with Go and Kafka

By Haseeb Majid

Elevator Pitch

Event-Driven Architectures (EDAs) whilst popular, can sometimes be hard to understand. In this talk I will go over what are the main advantages of EDA. How we can go about implementing services which leverage EDAs. Also why EDAs are not a silver bullet.

Description

Background

Lots of developers are building services with a micro-services architecture. However, there are some gotcha we need to be careful of, as with everything in software engineering micro-services are not a silver bullet.

Event-Driven Architectures (EDAs) can help solve some of the communication issues that can occur when building micro-services. Yet they can pose challenges for developers transitioning from synchronous (HTTP) systems/services. It is not always clear what the advantages are of EDAs. This talk aims to bridge the gap, highlighting the main advantages of EDAs in decoupling services.

Contents

In this talk, I will go over what EDAs are, and how we can use them to decouple our services. We will look at how EDAs allow our services to be asynchronous and contrast that against synchronous architectures such as a normal HTTP web service.

Kafka is one such tool used to help build EDAs, we will briefly look at how Kafka fits into EDAs. This will be the main tool we use to build our EDA with our Go service.

After covering some of the key concepts, we will look at two simple Go services, one which produces events and another which consumes those events. As we discuss the various EDA patterns we will look at how we can implement them in our Go services. Then we will look at how we can integrate Kafka with Go to subscribe and publish events.

Finally, we look at some pitfalls that can occur in an EDA.

Contents of talks

We will cover the following topics:

  • What are EDAs
  • Why use EDAs
  • What is Kafka
  • Why use Kafka
  • How can we use Kafka with Golang
  • Some common EDA patterns
    • Implemented in Go
  • Some EDA pitfalls

Notes

I think this would make a great talk because EDAs are becoming a lot more popular and I interact with a good number of developers who still don’t fully understand how they work.

So I think a talk explaining why use them and how to go about implementing from a service POV would be useful. There are tons of great resources online talking about EDAs, I want to focus a bit more how we could implement it in Go specifically.

I am very passoinate about EDAs, I’ve been on the receving end of building mico-services without using an EDA and it can be very painful, especially when things go wrong. Or getting stuck in HTTP call hell (service calls, a service which calls another service). So I’d love to help others avoid making the same mistakes I did.

Just to clarify I wont be doing any live coding, I intend to describe how a pattern works and why we might use it (such as outbox pattern). Then show a bit of code and how we might implement in a simple Go service.

I’m also open to simplifying/adjusting contents of the talk, as it does seem I’m covering a lot of stuff. My general style of talk is to introduce some concepts but not go super deep in any single concept. Leave that for the audience to go and look into in their own time. Its more of a what is possible with EDAs, if that makes sense.