Kafka meets Functional Programming.

The talk is on “Kafka meets FP”, with the ultimate intention of proving the fact that “Nothing actually stops us from striving to write a good piece of software”, and explain how Functional Programming in Scala can make us achieve it.

Having worked for various clients as a software consultant, it has always been my responsibility to write and make write good software - program that is type-driven, purely functional and maintainable. With this talk, we intend to prove that, regardless of the complexities and impurities that we have to deal when implementing a use-case, the software that we expose to clients can still be made reasonable, testable, deterministic and typesafe. This is done by taking a Kafka streaming application as the usecase.

As Kafka becomes a powerful stack for various companies to solve their business use cases, I believe it is important to talk about how Scala and FP can make us write good software that works with Kafka. Hence I am titling my talk to be “Kafka meets Functional Programming”.

Starting from listing down some of the runtime complexities and consequent challenges that we faced when playing with Kafka, I will describe a part of the use case we implemented for an insurance company, where we wanted to abstract out a Kafka streams application that can work for multiple mobility datasets.

After listing down the challenges and then describing a streaming use-case, we will start with an initial version of the solution written in Scala with a few type class constraints. Then, we will list down a bunch of issues with the code, motivating us to further improve. The second stage starts with type class that deals with returning a Kafka Serde, given a set of Kafka configurations, and then define an instance of Invariant functor instance for the same. Then we go step by step towards a final code base listing down the issues motivating us to improve the code at every stage. I won’t forget to summarise what we achieved once we finish with the final code base that is exposed to the users who don’t need to worry about Kafka details at all for new datasets.

Next, we will focus on writing test cases in a Kafka related application. This is important when we do a few deliberate mutations and untyped code-base to improve performance in a hot loop. This part of the talk intends to prove that, “If bad things are done for reasons, make sure we do good things in the test package.” I take Avro’s Generic Record as an example, focus on writing a test with the same, expose a test case that ended up in not working at all due to impure functions, and refactor it to work with that any F that can form a MonadState, and compare both solutions. There will be a point where we discuss “start writing pure functions instead of impurities and refactoring”. We finally make the higher kind F to be a scala-check.Gen to make it a property-based test that further improves the confidence level of the code. Again, we will reiterate through what we achieved by writing such a code.

Before concluding, I will be showing a snippet of tagless final code that operates on Schema Registry of confluent Kafka Serialization. This is part of an automation code base which we have written for a client to streamline a data pipeline with various other Kafka components.

Finally, we will reiterate through all that we have achieved by pushing ourselves hard writing FP with Kafka, we conclude with the fact that “nothing actually stops us from striving to write a good piece of software.”