From delayed polling to Real-Time Telematics with RabbitMQ and Erlang

Lucid Logistics, a subsidiary of Quake Global, provides end-to-end solutions for global GPS monitoring, management, and tracking of remote assets via the Iridium Low Earth Orbit (LEO) satellite network. One of the biggest challenges faced by Lucid Logistics has been distributing data immediately to customers as it arrives from the satellite provider. Historically data has been stored in a backend database and made available to customers in a somewhat generic fashion via user interfaces, email notifications, and reports. However, more customers now want to receive raw data feeds for their assets in near-real-time that they can process and act upon using their own specific set of business rules in order to best optimize their particular unique business operations.

Exposing the data for customers to consume via a web service was not a viable solution as this would require customers to poll for new data on some schedule, which would not only make inefficient use of resources but would also fail to address the real-time data requirement. Instead, extending the existing solution to better meet the demand for near-real-time data feeds required some type of publish/subscribe mechanism that allowed customers to subscribe to data feeds and would push data to subscribers in real-time as it was received from the satellite provider. Additionally, the solution needed to utilize protocols that could be readily accommodated by all customers.

These requirements led Lucid Logistics to PubSubHubBub, an open webhook-based protocol for distributed publish/subscribe communication over HTTP(s), initially designed to extend the Atom and RSS protocols for data feeds. This in turn led Lucid Logistics to RabbitHub, a powerful Erlang-based implementation of the PubSubHubBub protocol that operates as a RabbitMQ plugin. Not only was RabbitHub ideally suited to Lucid Logistics' needs in terms of providing the necessary protocol support, but because it is implemented as a RabbitMQ plugin, the resultant solution is also able to leverage and benefit from a myriad of RabbitMQ and Erlang features, including built-in security mechanisms, SSL support, message and message queue persistence, flexible routing topologies via exchanges, scalability and performance, and fault-tolerance through clustering.

In this talk the speakers will discuss how RabbitHub, RabbitMQ, and Erlang have been used to enhance the Lucid Logistics solution to provide customers with data in near-real-time. The PubSubHubBub protocol will be discussed and the operation of the RabbitHub plugin will be examined in the context of how it leverages various Erlang/OTP features and the RabbitMQ plugin architecture to provide a simple yet powerful HTTP-based interface to RabbitMQ's messaging infrastructure. The solution affords considerable flexibility and provides numerous opportunities for additional development of the Lucid Logistics message processing ecosystem, including providing data to customers via other protocols supported by other RabbitMQ plugins (MQTT, AMQP, STOMP, Web Sockets), and closer integration with the old database-based solution. These and other ideas will be briefly discussed.

Talk Objectives

To illustrate how RabbitHUB, RabbitMQ, and Erlang have been successfully used by Lucid Logistics to implement an elegant solution to deliver near-real-time data feeds to customers via the Internet in an efficient, reliable, and scalable manner.

Target Audience

General audience (there will be some content of a technical nature, but this will not be integral to the talk). Relevant to people interested in leveraging Erlang and RabbitMQ to provide real-time data delivery over the web in a secure, efficient, and scalable manner.

Slides
Video

About Natalya

Natalya Arbit is a senior developer with close to 15 years of professional experience spanning a variety of industries. She has spent the last 6 years working in telematics with a focus on supporting SaaS multi-tenant web portals using the Microsoft stack of technologies. She holds a Master of Science degree in Geology from University of Minnesota, Duluth, and a Bachelor of Science degree in Economics from the Russian State Geological Prospecting University.

Back to conference page