From delayed polling to Real-Time Telematics with RabbitMQ and Erlang

Lucid Logistics, a subsidiary of Quake Global, provides end-to-end solutions for global GPS monitoring, management, and tracking of remote assets via the Iridium Low Earth Orbit (LEO) satellite network. One of the biggest challenges faced by Lucid Logistics has been distributing data immediately to customers as it arrives from the satellite provider. Historically data has been stored in a backend database and made available to customers in a somewhat generic fashion via user interfaces, email notifications, and reports. However, more customers now want to receive raw data feeds for their assets in near-real-time that they can process and act upon using their own specific set of business rules in order to best optimize their particular unique business operations.

Exposing the data for customers to consume via a web service was not a viable solution as this would require customers to poll for new data on some schedule, which would not only make inefficient use of resources but would also fail to address the real-time data requirement. Instead, extending the existing solution to better meet the demand for near-real-time data feeds required some type of publish/subscribe mechanism that allowed customers to subscribe to data feeds and would push data to subscribers in real-time as it was received from the satellite provider. Additionally, the solution needed to utilize protocols that could be readily accommodated by all customers.

These requirements led Lucid Logistics to PubSubHubBub, an open webhook-based protocol for distributed publish/subscribe communication over HTTP(s), initially designed to extend the Atom and RSS protocols for data feeds. This in turn led Lucid Logistics to RabbitHub, a powerful Erlang-based implementation of the PubSubHubBub protocol that operates as a RabbitMQ plugin. Not only was RabbitHub ideally suited to Lucid Logistics' needs in terms of providing the necessary protocol support, but because it is implemented as a RabbitMQ plugin, the resultant solution is also able to leverage and benefit from a myriad of RabbitMQ and Erlang features, including built-in security mechanisms, SSL support, message and message queue persistence, flexible routing topologies via exchanges, scalability and performance, and fault-tolerance through clustering.

In this talk the speakers will discuss how RabbitHub, RabbitMQ, and Erlang have been used to enhance the Lucid Logistics solution to provide customers with data in near-real-time. The PubSubHubBub protocol will be discussed and the operation of the RabbitHub plugin will be examined in the context of how it leverages various Erlang/OTP features and the RabbitMQ plugin architecture to provide a simple yet powerful HTTP-based interface to RabbitMQ's messaging infrastructure. The solution affords considerable flexibility and provides numerous opportunities for additional development of the Lucid Logistics message processing ecosystem, including providing data to customers via other protocols supported by other RabbitMQ plugins (MQTT, AMQP, STOMP, Web Sockets), and closer integration with the old database-based solution. These and other ideas will be briefly discussed.

Talk Objectives

To illustrate how RabbitHUB, RabbitMQ, and Erlang have been successfully used by Lucid Logistics to implement an elegant solution to deliver near-real-time data feeds to customers via the Internet in an efficient, reliable, and scalable manner.

Target Audience

General audience (there will be some content of a technical nature, but this will not be integral to the talk). Relevant to people interested in leveraging Erlang and RabbitMQ to provide real-time data delivery over the web in a secure, efficient, and scalable manner.

Slides
Video

About Brett

Brett Cameron works as a senior technical consultant at VMS Software Inc. (VSI, http://www.vmssoftware.com/index.html), helping to define and implement the company’s Open Source strategy for the OpenVMS operating system. Before joining VSI Brett worked as a senior software architect with HP’s Cloud and Enterprise Services groups. Brett lives in Christchurch, New Zealand and has worked in the software industry since 1992, and in that time he has gained experience in a wide range of technologies, many of which have long since been retired to the software scrapheap of dubious ideas. Over the past decade Brett has spent considerable time travelling the world helping organisations to modernize their legacy application environments and to better leverage Open Source technologies. In more recent times, his involvement with various Open Source projects and his work in the cloud computing space has caused him to develop a liking for functional programming languages, and Erlang in particular, which he has ported to several operating systems, including OpenVMS. Brett holds a doctorate in chemical physics from the University of Canterbury, and maintains close links with the University, delivering guest lectures and acting as an advisor to the Computer Science and Electronic and Computer Engineering departments on course structure and content. In his spare time Brett enjoys listening to music, playing the guitar, and drinking beer.

Twitter: @brc859844

Github: brc859844

Back to conference page