Push Jobs: A Scalable Remote Execution System for Chef

When Chef needed a scalable system for remote job execution on thousands of systems simultaneously, we chose Erlang and ZeroMQ for the job. This talk will cover: 

* How we applied the 'let it crash' philosophy across a system of thousands of nodes.

* The various impedance mismatches between the ZeroMQg and Erlang approaches to distributed systems.

* The various bottlenecks we hit as we scaled our system, and how we overcame them.

Talk objectives:

- To describe our experiences scaling an zeromq application written in erlang, along with our missteps and unexpected successes.

Target audience:

- Anyone who is interested in building systems with ZeroMQ and Erlang

 

Slides
Video

About Mark

Mark Anderson is a Software Engineer on the Community Team at Chef (formerly Opscode), working on building scalable server architectures using Erlang and building open source communities around them. Before Chef he was at Microsoft, first on the Binary Analysis group at Microsoft Research and then on the early days of the Azure project.

Github: manderson26

Back to conference page