Tracing and debugging large systems still remains a challenging and sometimes daunting task. There are few tools at hand, and even if some data is collected the sheer volume and hours required to analyse it makes one wince. However, there's a hidden gem in OTP which many developers and testers may not be aware of - the Inviso application, providing support for setting up and collecting real-time tracing in multi-node environments. Although still a work in progress, we've made an effort to provide a clean, functional and safe API to it. Together with a friendlier user-interface it is getting ready to assist developers who are building massively distributed systems.

But on-line tracing is one thing; post-mortem debugging is another. As a developer or support engineer you have probably spent enormous energy and time manually analyzing log files, trying to find out what went wrong, when and why.

Using Exago, now you can automatically parse and process log files, and check them against an abstract model of the system. In case of failure, it will report the abstract state where the error occurred, and the events that led to the point of failure.

In brief, this tool automates most of the difficult and tiresome task of manually inspecting log files, tasks that you would probably pass to the interns.