Building a Distributed Data Ingestion System with RabbitMQ

Your company has servers distributed around the world and you need to process data in a centralised location. The data is produced by applications using different technology stacks and comes from various sources, from web servers to sensors. How could you solve this problem? Enter RabbitMQ. In this talk we are going to show how to build a system that can ingest data produced at separate geo located area (think AWS and it's many regions) and replicate it to a central cluster where it can be further processed and analysed. We will present an example of how to build a system like this one by using RabbitMQ Federation to replicate data across AWS Regions and RabbitMQ support for many protocols to produce/consume data. To help with scalability we are going to show an interesting way to implement sharded queues with RabbitMQ by using the Consistent Hash Exchange. If you want to learn what else has RabbitMQ to offer beyond simple messaging and queueing, then this is the talk for you.

Slides
Video

Tutorial: Scaling Applications With RabbitMQ

Do you need to process thousands of images in the background for your web app?

Do you need to share data across multiple applications, probably written in different languages and sitting at different servers?

Your real time data feed is becoming slow because you are polling the database constantly for new data updates?

Do you need to scale information processing during peek times?

What about deploying new features with zero downtime? If any of these problems sound familiar then you probably need to use messaging in your application.

In this tutorial I will introduce RabbitMQ, a messaging and queue server that can help us tackle those problems. We will learn the benefits of a Queue Server and see how to integrate messaging into our applications. With this tutorial we hope that the term 'decoupling' gets a new, broader, meaning.

Tutorial objectives:

- Learn the benefits of messaging oriented architectures
- Learn about RabbitMQ
- Learn about several messaging patterns that can help you scale our apps

Target audience:

A beginner to intermediate audience that wants to learn how to use
RabbitMQ to solve scalability and interoperability problems.

REGISTER HERE

About Alvaro

Alvaro Videla used to work as Core Developer for RabbitMQ. Before moving to Europe he was working in Shanghai where he helped building one of Germany biggest dating websites. He co-authored the book "RabbitMQ in Action" for Manning Publishing. Some of his open source projects can be found here: http://github.com/videlalvaro. Apart from code related stuff he likes traveling with his wife, listening/playing music and reading books.

GitHub: videlalvaro

Twitter: @old_sound

Back to conference page