Stop Guessing and Start Measuring - Benchmarking in Practice

“What’s the fastest way of doing this?” - you might ask yourself during development. Sure, you can guess, your intuition might be correct - but how do you know? How long would that function take with a million elements? Is that tail-recursive function always faster?

Benchmarking is here to give you the answers, but there are many pitfalls in setting up a good benchmark and analyzing the results. This talk will guide you through, introduce best practices, and surprise you with some unexpected benchmarking results along the way. You didn’t think that the order of arguments could influence its performance...or did you?

Talk objectives:

The talk aims to build a general understanding of benchmarking and best practices in benchmarking to avoid common pitfalls and wrong results as often observed in the real world.

The talk marries the explanation of the general concepts with some practical usage and examples with the benchee library for benchmarking (https://github.com/PragTob/benchee) that I'm the author of. Examples also include some surprising results, like how body-recursive functions can be faster than tail-recursive functions, how the order of arguments can affect performance on the BEAM, how much slower Map.merge/3 is compared to merge/2 etc.

A slight introduction into statistics is included to make sense of the data and show which values are important where, what they are good for and why "average" isn't the best value and why one might prefer median/nth percentile

The library is written in Elixir, but I always wanted to make it work with Erlang so I vow that if accepted I'll look into what improvements I can make for easy usage from Erlang :)

In short:

• distinction micro/macro/application level benchmarks and use cases for them
• basics and best practices of benchmarking
• practical design of benchmarks with the example of benchee (with real world benchmarking examples from our applications)

Target audience:

• Everyone who works on a system with some performance critical parts that they want to improve or anyone who is performance minded. I know that even people rather familiar with Benchmarking can learn something here. Also people who are just in for the "Whaaat, the order of arguments can have a signifcant performance impact, woot" moments. 

Slides
Video

About Tobias

Tobi is a clean coder, Full Stack developer, Rubyist, Elixir fan, learner, teacher and agile craftsman by passion. He organizes the Ruby User Group Berlin, maintains Shoes and Benchee as well as contributing to a variety of projects while thinking about new ideas to put into code and push boundaries. He loves collaboratively creating just about anything people enjoy. Currently he's creating wonderful web applications, most recently with Elixir and Phoenix refreshing his perspective on web applications.

GitHub: PragTob

Twitter: @PragTob

Back to conference page