Kafkerl: A Fault Tolerant Logging Library For Kafka

A reliable logging system is a must have when working on a project that requires statistics and audits. Due to its distributed and fairly reliable nature using kafka is a great solution. But we need to have a good way to communicate with it, not only using the right protocol but also dealing with any downtimes without losing a single message. To achieve this, we made Kafkerl, a library that not only handles bursts of messages with ease but that also avoids losing any information by using a combination of local files and ETS tables. In this talk I will show you how Kafkerl was built and highlight the way it handles these issues, so you can also use it in your projects whenever you find yourself facing the same requirements.

Talk objectives:

* Increasing throughput through the use of ETS tables.

* Thinking about reliability from the get go.

* Designing for failure.

Target audience:

* Anyone not wanting to lose important data when the unexpected happens.

* Kafka users.

* Library builders.

Slides
Video

About Hernán

Videogame programmer turned Erlanger after making a game server with Erlang and falling in love with functional programming.

Hernán is currently working at Inaka as a developer and his hobbies include dialyzing Erlang by hand and complaining about technology.

GitHub: hernanrivasacosta

Twitter: @HernanInaka

Back to conference page