Maestro - Orchestrating Large Scale Multiplayer Games

This talk is about our journey of going from monolithic game servers to component based game services. We will present the Maestro Platform that consists in a multi-component architecture, developed in Erlang, which aims to orchestrate an infrastructure of game servers. When faced with the requirement of building large scale real-time games, and considered our experiences with our past products, we came to the inevitable conclusion that we had to rethink the architecture of our game servers and the services to support them. That’s when the Maestro platform was born as a central piece development and deployment of new games (mobile and web).

The main goal of Maestro is to accommodate all our knowledge and technologies as pluggable components that can be combined based on the specific requirements of each game, as well as their acceptance in the market. We started our design by guaranteeing that the platform would help us automatically scale our game server infrastructure based on rules defined by the game itself (to achieve scalability and availability). The Maestro platform will be able to launch new instances of game logic arenas/rooms and monitor their health and status, perform scaling decisions, or even recover from unexpected crashes, by relaunching these managed arenas as a erlang supervisor would do with any gen_server. 

We separated the game play logic from the service logic in a way that our client development teams can program also the server side game play logic in their preferred language (mainly C++) almost as a black box. To accomplish this we use a defined high level protobuf protocol for client and for the game server logic and also a well defined API, both of them providing extensibility points for use on a game-by-game basis. 

The services layer provides components/services to manage the business logic (IAP’s, Analytics, Accounts) and also common used features support like Leaderboards, Matchmaking, Gifting, Social Networks and new ones that can be reused in other titles. 

This approach lead us to cutting down on re-implementing solutions to simply re-use and reapply. We are able to launch a global scale infrastructure for a new game with integration included within just one week of coordination and integration between game client and backend developers. This also give us room to improve a single product and concentrate our energy in improving only one “product” but affecting all the current and future products.

Talk objectives:

• Discuss the challenges, benefits of going from a monolithic server design to a component based service architecture
• Discuss how we rely on erlang to give us the scalability, resilience and availability we need for massive multiplayer online games
• Discuss some design patterns and architecture decision we have adopted 

Target audience:

• People interested in large scale deployment of erlang software servers 
• Erlang services architects and developers 
• Game industry backend developers

Slides

About Pedro

Pedro was a military (rank Captain) senior software engineer working in research and development of UAV technologies targeting embedded systems to implement cooperative (distributed) UAV maritime surveillance missions. Currently he works at Miniclip were he started as backend developer of game servers in Java and backend services in Erlang, in the past 1 year and half he was dedicated only to erlang development, becoming a Lead Engineer for all game services development team, where the main languages are Erlang and C++. Pedro is really interested and an enthusiast of distributed systems architectures and concepts. He likes to apply theoretical concepts to solve “real world” challenging problems.

GitHub: posilva

Twitter: @pmosilva

Back to conference page