Erjang - A JVM-based Erlang VM
| Over the last few years, I have been meeting "Erlang people" more and more often, and I was getting this clear impression that "you people" have some kind of magic ability to reason intuitively about concurrent systems in a way that I could not. That bothered me, so I wanted to learn Erlang. Being a language implementer, the most obvious way to do that is to just go ahead and implement an Erlang VM, right? The result of this "little exercise" is Erjang, an open-source JVM-based Erlang VM, which I think has potential to be useful as a means to push Erlang into new areas of application, help grow the Erlang community, and most importantly make all those Java-heads out there willing to learn how to do concurrency right. In this "brave new world" of networked services and multi-core everywhere, there are a lot of people who are challenged in providing even basic reliability, efficiency and correctness. My hope is that Erjang's existence can influence some of those outside the Erlang community to do better. In technical terms, Erjang reads .beam files and compiles them to Java's equivalent .class files which are then read into the running JVM. It runs off a plain Erlang/OTP distribution - it only requires the beam files from there; Erjang itself is written in Java. As off this writing, it can run some non-trivial erlang programs, but is not yet capable of booting OTP [will have to update this abstract as we go along - follow updates on my blog]. Comparing the BEAM virtual machine and Erjang, the most obvious differences are that (a) Erjang will not be able to provide [soft] real-time guarantees since it uses Java's garbage collector, and (b) it has limited support for native code and port drivers (other than file and network I/O). The upside is the new ways this allows us to deploy Erlang systems. In this presentation I will talk about how Erjang works, what behaviour to expect from Erlang programs running on top of Erjang, what I learned along the way, and demonstrate [live] that Erjang runs well enough to be obviously useful.
|
| Cloudi is a free Erlang based private cloud for efficient processing in C or C++ to maximize hardware utilization with dynamic load balancing. Cloudi relies on external databases for keeping the work fault-tolerant by preserving the work data. Implementing work for the cloud is as simple as declaring the cloud interface. The presentation provides an introduction to the Cloudi framework.
|
A discussion of basic vs. applied research in the software domain and the creation of Erlang
| This talk will present some of the thinking behind the creation of the Computer Science Laboratory (CS Lab) at Ericsson - one of the technology oriented labs within Ericsson Research. It is at the CS Lab where a group effort in software research resulted in Erlang's birth. The talk will describe some of the projects which lead to Erlang, and when it finally became mainstream within Ericsson, the OTP Erlang middleware project. It will conclude by describing how the technology transfer to the newly created Erlang/OTP unit happened.
|
Achieving Parsing Sanity with Neotoma
| Most developers will be familiar with lex, flex, yacc, bison, ANTLR, and other related tools to generate parsers for use inside their own code. Erlang has its own pair, leex and yecc, for accomplishing most complicated text-processing tasks. For computer-friendly languages, however, context-free grammars and their parser-generators leave a few things to be desired. This talk is about how the seemingly simple prospect of parsing some text turned into a new parser toolkit for Erlang called Neotoma, and why functional programming makes parsing fun and awesome.
|
| For a long time, and due to the lack of main FP concepts in most mainstream languages, we missed opportunities to abstraction and code expressiveness and conciseness. With today’s democratization of FP, Computational Abstraction is what will enable us to be less dependent on specific programming language syntax offering; creating libraries of control structures and composition forms that help find concise and expressive solutions to enterprise programming challenges (null, lists treatment, error handling), capturing elegantly important business concepts in code, and programming at the right level of abstraction. For a long time, and due to the lack of main FP concepts in most mainstream languages, we missed opportunities to abstraction and code expressiveness and conciseness. With today’s democratization of FP, Computational Abstraction is what will enable us to be less dependent on specific programming language syntax offering; creating libraries of control structures and composition forms that help find concise and expressive solutions to enterprise programming challenges (null, lists treatment, error handling), capturing elegantly important business concepts in code, and programming at the right level of abstraction.
|
Deploying Erlang into a Large Organization - A Case Study
| We have been using Erlang on our team at Yahoo! for the last 2 years, during which time we have been advocating for its official support. While it is easier, and possibly more fun, to be able to mandate Erlang as the prefered technology, it is more realistic to be confronted with complex integration scenarios. In this talk we will present several case studies of integrations we have done as well as the challenges we have faced in justifying our choices. In order to deploy Erlang effectively we have had to interoperate with company standard stacks, as well as various legacy systems in written in various other languages.
|
Distributed Erlang Systems In Operation: Patterns and Pitfalls
| While Erlang/OTP provides a rich set of abstractions and tools for developing distributed applications, the operation of a distributed Erlang system at scale presents many challenges and pitfalls that developers should be aware of at all phases of application development. In this talk, Andy will present a number of techniques for building operationally friendly Erlang systems that effectively use the OTP libraries, based on real-world experience at Mochi Media and Basho Technologies.
|
Enet: TCP/IP in Pure(ish) Erlang
| Erlang is great for implementing network protocols, so let's write a TCPIP stack top to bottom. The Enet library includes a port program for accessing an OS TAP device, a suite of protocol encoders and decoders and a network interface module that correctly responds to Ping. This talk covers a simple port program, How the binary syntax saved my hard-drive, tcpdump in Erlang, grandiose ambition, and the highest ping times you've ever seen over a loopback adapter.
|
Erlang Is Our Superpower: How Collecta Uses Erlang
| Collecta is a real-time search engine that allows users and developers to access enormous of amounts of information at extremely low latencies. Our system is almost entirely Erlang powered, making use of ejabberd, Webmachine, CouchDB, Riak, RabbitMQ, as well as our own Erlang projects, and is deployed over more than a hundred machines.
I will talk about why Collecta choose to focus our development on the Erlang language and why our problem domain is well suited to Erlang's strengths. Search encompasses a large number of problems, but we find that Erlang excels at solving many of them. I'll also describe our general architecture as well as the reasoning behind several of our technology choices.
|
Erlang SMP support - behind the scenes
| This talk will tell you the story about how Erlang got multicore support and will give you all the gory details about utilizing multicore processors in a conventional programming language. I'll tell you what we've done at OTP so that you, as an Erlang programmer, can sit back and enjoy the fact that you don't have to bother with such things!
|
| This talk will cover the unique aspects of Erlang that are now the basis of a highly scalable web services platform at E*TRADE Financial. It will cover the criteria used in selecting Erlang over other approaches, the design aspects of the current system as well as issues that were encountered and addressed during the development, testing and deployment of the system. The 'Common Web Services' framework is an E*TRADE Technology for advertising, monitoring and routing web service calls in a distributed system consisting of C/C++ and Java service implementations. The framework allows services to be dynamically advertised, called from synchronous and asynchronous clients as well as forwarded. Each service call may be queued if the back end pool of workers is busy, and all services and workers can be monitored and controlled via a command line or web interface. |
Erlang secure RPC and the SSH module
| Remote Procedure Call (RPC) over public Internet has to be cryptographically protected for secure communication. In the current Distributed Erlang, this cannot be done in a firewall-friendly way. This presentation provides an introduction of writing code using Secure Shell (SSH) protocol on Erlang with Erlang/OTP ssh module, and a description of an RPC method based on SSH subsystem implemented on Erlang.
|
Extending ejabberd with Google Wave support: Adding realtime collaboration to realtime messaging.
| We have developed our ProcessOne our own interoperable version of Google Wave server. It supports a large subset of the required feature (From Wave operational transformation engine to Wave storage). The default client protocol is XMPP, meaning our Wave server can work as an ejabberd XMPP server extension and can be use directly inside XMPP client.This talk will present the Wave protocol, our Wave server, developed in Erlang, and its architecture.
|
| While it is not a fast language, for many purposes Erlang is considered "fast enough". But what happens when "fast enough" isn't fast enough? You don't have to rewrite your whole project in C or C++. This talk will cover the exercise of profiling an Erlang application. You will learn how to replace critical code sections with C functions using the new NIF (Native Implemented Function) interface. You will learn how to integrate a native 3rd party library using the linked in driver interface. And finally you will learn how to protect C drivers from race conditions in Erlang's highly concurrent environment and how to debug them in a running Erlang VM.
|
High Availability in the Cloud
| The talk will focus on how I (with the help of the entire Cloudant
team) built our database service based on CouchDB on top of EC2.
Specifically how we use Erlang, Chef, EC2 and other tools to build
highly available and performant database clusters. This includes using
Chef and Erlang's hot code upgrades to automate cluster-wide upgrades
without restarting any services.
The main topics will include: 1. Getting high availability in the cloud
|
Inviso and Exago: tracing and log analysis in multiple node environments
| Tracing and debugging large systems still remains a challenging and sometimes daunting task. There are few tools at hand, and even if some data is collected the sheer volume and hours required to analyse it makes one wince. However, there's a hidden gem in OTP which many developers and testers may not be aware of - the Inviso application, providing support for setting up and collecting real-time tracing in multi-node environments. Although still a work in progress, we've made an effort to provide a clean, functional and safe API to it. Together with a friendlier user-interface it is getting ready to assist developers who are building massively distributed systems. But on-line tracing is one thing; post-mortem debugging is another. As a developer or support engineer you have probably spent enormous energy and time manually analyzing log files, trying to find out what went wrong, when and why. Using Exago, now you can automatically parse and process log files, and check them against an abstract model of the system. In case of failure, it will report the abstract state where the error occurred, and the events that led to the point of failure. In brief, this tool automates most of the difficult and tiresome task of manually inspecting log files, tasks that you would probably pass to the interns.
|
JavaScript CouchApps with CouchDB
| I'll discuss the projects goals, adoption, and inner workings of the popular document database. I'll show some JavaScript code, for those of you who'd like to Relax and learn to write offline capable, peer replicating CouchApps.
|
Latest news from the Erlang/OTP team at Ericsson
| Kenneth gives an update of the Erlang/OTP team's work at Ericsson - their current projects and plans for future.
|
| There is growing interest in embedded languages inside the Erlang platform, from erlang_js to erl-lua. As a fan of embedded languages, Chad's talk will explore the process of integrating Lua with Erlang as a built-in driver; calling into Lua from Erlang and vice versa. We will do this through exploration of the creation of a distributed UNIX systems management utility created in Erlang and Lua. Most frameworks for process, file, and server monitoring treat distributed monitoring across multiple systems as an afterthought. Erlmon is built from the beginning to allow distributed monitoring of UNIX systems, all in Erlang. But the average system administrator doesn't want to learn Erlang syntax to monitor services in the network, which is where Lua comes in. We'll explore the creation of the erlmon utility, with a focus on the integration of the configuration system in Lua, and we'll discuss a case study of a real-world system using this monitoring tool.
|
| Git is the distributed version control system that's taking the world by storm. The Linux kernel, Ruby on Rails, and now even Erlang/OTP have all chosen it as their VCS of choice. This talk will focus on mastering the handful of commands that comprise 90% of the functionality you will use on a day-to-day basis. I'll kick things off with the best method to install Git on your favorite platform. Next up is creating local repositories, staging/adding files (all about the index), making commits, and viewing history. From there you'll learn to share your repositories with others via GitHub, fetch and merge changes that others make, and resolve conflicts. I'll also introduce a few common workflows that you can use to streamline collaborative development. Each step of the way I'll demonstrate the commands and explain what's happening under the hood to help you establish a proper mental model that will enable you to master the more complex parts of Git with ease. Even if you already know Git, there should be something new for you in this talk. By the end of the session you will be well on your way to using Git to make your life easier than you ever thought possible!
|
Power debugging C code with Erlang
| QuickCheck tests stateful code beautifully, generating short sequences of API calls that reveal broken behaviour. Great for testing Erlang code... but equally useful for other programming languages, provided we can invoke them seamlessly from Erlang. John will demonstrate Quviq's new link to C, which relieves the user from writing any boiler-plate code whatsoever, making property-driven development in C, with the properties written in Erlang, truly practical.
|
| Riak Search is a distributed data indexing and search platform built on top of Riak. The talk will introduce Riak Search, covering overall goals, architecture, and core functionality, with specific focus on how Erlang is used to manage and execute an ever-changing population of ad hoc query processes.
|
SCREAM: Static Analysis of Regular Expressions for Analysis and Modifications
| This paper illustrates an interesting application of Erlang; specifically, one dealing with analysis of encoded data in a static context. The root problem is one of analysis of data streams. In many cases, devices may wish to monitor streams of data for interesting patterns, but such analysis engines may be limited in the complexity of operations supported for such analysis. A practical example of this is a common one: a network intrusion detection system may wish to analyze email messages without having to store and forward each message. Many email systems encode binary data using the Base64 transform, a bitwise encoding scheme. For performance reasons, it is sometimes desirable to not first decode the message before analyzing its contents. This paper presents a tool, b64re, that analyzes a regular expression and transforms it such that it will now match its input when said input has been encoded using Base64. Several features of Erlang/OTP are illustrated, including parsing, the ease with which bitwise data can be manipulated, using multiple distributed processes to speed calculation, and the use of Erlang as a language in contexts other than distributed, soft real-time applications.
|
Scala for the Curious Erlang Programmer
| Scala is a statically-typed, hybrid functional and object-oriented language for the JVM. The Scala standard library includes an Erlang- inspired Actors library. In this talk, I'll discuss how Scala compares and contrasts to Erlang, highlighting the advantages and disadvantages of each language for particular needs. For example, we'll discuss the pros and cons of a rich type system and static typing in Scala. We'll discuss ways that Scala is perhaps more general purpose than Erlang, but not as powerful in the areas where Erlang excels.
|
Testing abstract data structures with QuickCheck
| In this presentation we present a method for testing library modules that implement data structures. The method is applicable to OTP libraries like sets, dict, queue etc, but also to your own libraries. The method is based on writing a data type specification in Erlang and using QuickCheck to generate test cases. The method guarantees that the data structure is fully tested and that no other tests need to be added.
|
| One year ago, Joel Reymont challenged the Erlang community to build an open source stock market. In this talk, you'll see the result of my effort to build it - the triumphs, the frustrations, and the benchmarks; and I will directly compare the Erlang solution to a Java solution we built for a client.
|
The good code, the bad code and the ugly code: using Wrangler to refactor Erlang programs and tests
| Wrangler is an interactive refactoring tool for Erlang. We will introduce the system through a series of short demonstrations, and then concentrate on the 'similar code' detection facilities of Wrangler. Using these with the refactorings in Wrangler we show how code - particularly test code - can be shrunk dramatically, under the guidance of the test engineer. The talk is illustrated with examples from Open Source and commercial Erlang development projects. |
The nine nines and how to get there
| Once upon a time the flagship product of the Erlang world was the AXD301, an phone switch from Ericsson sporting 1-2 million lines of highly complex Erlang code. This product was supposedly capable of achieving service availability levels of "nine nines", i.e. 99.9999999% reliability. This is, even by telecom standards, insanely high. In this talk, you will hear about this, what was actually achieved, and how it was done. The talk will not be about software architecture, testing or "methods to ensure quality", but rather ruminations about debugging in general and the unique opportunities for power debugging offered by the Erlang virtual machine.
|
|
|
Using Erlang in a Carrier-Grade Media Distribution Switch
| For the past three years I've worked in a startup developing innovative hardware solutions to handle the staggering growth of video traffic not only on the Web, but also in traditional cable TV networks and in IPTV deployments. As network operators rapidly push toward providing seamless delivery across TV screens, computer screens, and mobile devices, they're finding that traditional approaches of racking and stacking commodity PCs are quickly running out of steam due to overcrowded and overheating data centers with insufficient power availability, inadequate media storage solutions, and limited network bandwidth. In this talk I'll detail how Erlang has delivered for us on its promise of providing a superior platform for highly concurrent, distributed, fault-tolerant, and long-running applications. Compared to traditional C and C++ approaches typically used for these types of systems, our Erlang applications provide many advantages, such as being far smaller and simpler, much more flexible and maintainable, easier and faster to build, debug, and test, and significantly more capable of utilizing multiple cores. I'll cover topics related to web applications, integration with specialized hardware video delivery capabilities and with non-Erlang applications, distributed systems solutions, testing, simulation, debugging, and deployment. If you're attending Erlang Factory because you're trying to determine if Erlang is right for your project, or if you're just relatively new to the language, note that I'll also discuss the journey from initially having to convince management that Erlang was even worth serious consideration to now having it serve as an indispensable part of our products. |
Webmail for millions, powered by Erlang
| How do you design and implement a custom, complex web-based email system that must be robust, scalable across dozens of machines, and support two million users at the start? The small size of your entire staff is a handicap. Everything must be tested to the rigorous standards of telecom giant in Asia which overlooks very, very few details. I'll describe one way to do it, using Erlang at its heart for: * Communicating with the Web browser client * Speaking SMTP, POP, and IMAP to the rest of the email world * Interfacing with a proprietary, external text index/search and user authentication services * Gluing C++ components for speed-critical and legacy feature support * Using a dynamically scalable key-value DB to store everything. All email, user profile, and vCard data is stored in a distributed key-value database that supports dynamic online resizing as the service grows while keeping email stored safely and consistently. We make heavy use of protocol contracts (using UBF) to enforce over-the-wire sanity of all intra-system communication and with the browser. It's tested extensively with EUnit, QuickCheck, and Python scripts. I'll give details of what has worked well and what we should have done differently through each stage of design, implementation, testing, and deployment.
|
| Francesco Cesarini, Technical Director of Erlang Solutions Ltd - organisers of the Erlang Factory - opens the event
with a short address.
|
What are the important ideas in Erlang?
| Erlang combines ideas from functional programming, logic programming, distributed systems construction, operating systems design and packages these into a platform that can be used for building fault-tolerant scalable systems. But where did these ideas come from? Which are the most important ideas? This talk presents a personal view of what I think are the important ideas in Erlang. I'll talk about where the ideas came from and how they developed and changed with time. I'll also talk about the failed ideas - the ideas we had that lived for a short while, even implemented, and then were removed from the language. The failed ideas are interesting because they often seem to be superficially correct, but on deeper inspection turn out to be wrong. I'll talk a little bit about some of the ideas which are "good but not yet implemented" usually these represent ideas where behind the scenes their lurks a horribly difficult implementation problem.
|
What is messaging and why should you care?
| Alexis will be presenting on RabbitMQ, an Erlang implementation of AMQP, the emerging standard for high performance enterprise messaging. Erlang is well known delivering a highly scalable and stable environment for applications that involve messaging. When JP Morgan Chase and others introduced AMQP, a new standard l4 protocol for busisness messaging and integration, Erlang seemed an obvious choice. This talk will explain the business rationale of this decision in detail and describe the technology and architecture of the ensuing product, RabbitMQ. Today, RabbitMQ is used in solutions across multiple languages and platforms such as Java, Linux, C# on .net, Python and of course Erlang. For people who want to benefit what Erlang brings to the table, products that implement a protocol as their API seem to be delivering value. |
| Ever thought about writing a book? I think many of us have, but have no idea where to start or what it would take. In this talk we will write our experiences over the last two years writing the book Erlang and OTP in Action for Manning. The journey has been a challenging, thought provoking, informative, and very often times an absurd and humorous one. We will relate everything from how togo about proposing a book to a publisher, coming up with a good table of contents, working with an editor, reviews, production and moving toward print, and quite a bit more. The presentation slides from this talk are available here: http://prezi.com/pmq-hxzhh7vv/writing-a-technical-book/
|
