Build big with tiny tools: immutability, checksums, and CRDTs

Designing, building, testing, and operating distributed system remains a hard problem because of complexity. What happens if we use only a handful of limited tools & techniques to build a distributed system?

For example:

1. write-once registers

2. cryptographic checksums

3. CRDTs: Conflict-free Replicated Data Types

This talk explores the architecture and implementation of Machi, a replicated file store that uses each tool above. First, each byte in a file is a write-once register. Second, every file byte is protected by client-calculated, end-to-end system checksum to detect file corruption. Third, all file contents can be merged in any order at any time without conflicts, via a CRDT-inspired design. Elsewhere, Machi uses write-once registers and CRDTs to orchestrate file data replication via Chain Replication. Together, these tiny tools create a system that is robust, very resistant to data loss, and easy to test.

Talk objectives:

* Starting with the constraints of our simple tools, show what a write-once, always mergeable file store looks like from a distributed systems point of view. We also show that this file store can provide either strong consistency or eventual consistency semantics (but not both at the same time!).

* Explore how a software developer can exploit Erlang's many strengths to implement a robust and scalable file store based on the foundation tools.

* Discuss the testing methods used to verify that our Erlang implementation behaves correctly.

Target audience:

Erlang developers who want to design safer applications with properties like, "A server never loses data," and/or need to test those apps, e.g., "What does 'never loses data' really mean?"

Slides
Video

About Scott

Scott Lystig Fritchie was a UNIX systems administrator until he returned to programming full-time at Sendmail, Inc.  While at Sendmail in 2000, a colleague introduced him to Erlang.  His world hasn't been the same since. In addition to write Erlang and occasionally C for the Erlang virtual machine, he has had papers published by USENIX, the Erlang User Conference, and the ACM and has given presentations at Erlang Factory and Ricon.  He is co-chair of the 2016 ACM Erlang Workshop (to be held in Nara, Japan) and will probably ask you to submit a paper for the workshop. Scott is a senior software engineer at Basho Japan.

GitHub: slfritchie

Twitter: @slfritchie

Back to conference page